{"id":"bae1eabf-2455-41ab-8bf5-ff4c34b3ef2a","arxiv_id":"2509.01620","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"A new modern Chinese poetry detection benchmark shows most current AI-text detectors are unreliable, particularly when LLMs imitate a human style.","lead":"This paper builds the first benchmark for detecting AI-generated modern Chinese poetry, pairing 800 human poems with 41,600 poems from four large language models under 13 different instruction types. It then evaluates six detectors and finds that most have near-chance accuracy, especially when AI poems imitate a human poet's style.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Reuse of the same 800 human poems across all conditions creates leakage risk in cross-feature results; a disjoint-poet test is needed.","rationale":"The paper's central negative claim—that current detectors cannot reliably detect LLM-generated modern Chinese poetry—depends on the AIGenPoetry human corpus being representative of the real target distribution. That assumption is weakest where the design is most convenient: the same 800 poems by six professional poets are reused as the human class in every prompt condition. The paper does not describe train/test splits, thresholds for zero-shot detectors, or any sensitivity check with a disjoint poet set, so the reader cannot determine whether the fine-tuned RoBERTa in Table 4 is memorizing the human poems from training. This is not an allegation of misconduct; the authors may have used standard random splits, but the documentation gap creates a concrete leakage risk. If leakage exists, the generalization claims and the conclusion that style is the hardest intrinsic quality could be artifacts of the experimental setup rather than properties of the poetry. The proposed disjoint-poet test would settle this. The paper has strengths—a carefully assembled dataset, a systematic prompt design, and additional temperature experiments that probe a plausible confound—but those do not remove the need to validate the human corpus. Since the concern is testable and the benchmark is still useful, the conditional verdict is appropriate; no change to the reader's verdict is needed.","tokens_in":23617,"tokens_out":10074,"duration_ms":105417,"concrete_test":"Run the Table 4 generalization experiments with a disjoint set of human poems for train and test: e.g., hold out all poems by one of the six poets during training (or split the 800 human poems so that no test-condition human poem appears in the training set), then retrain RoBERTa on D2+4+5 and evaluate on D1/D6-13. Compare the resulting F1 to Table 4. If RoBERTa's F1 drops by more than a few points, the original numbers were inflated by human-poem memorization. Additionally, rerun the D2/D3 style condition with this disjoint-poet protocol to see whether style-matched poems remain the hardest.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The benchmark's human class is the same 800 poems by six poets in every prompt-condition dataset Di (Section 4). In the generalization experiments (Table 4), a model trained on D2+4+5 or D6-8 has very likely already seen the identical human poems that appear in the test conditions (D1, D6-13, etc.). For the fine-tuned RoBERTa, the human side of the test set can then be solved by memorization rather than by detecting LLM-generated text, inflating F1 and confounding both the generalization conclusions and the relative difficulty ordering of styles vs. emotions. In-domain comparisons (Tables 2-3) are less directly corrupted because each Di is split internally, but the shared 800 poems still tie every condition to a single narrow human corpus. The paper does not report train/test splits or any held-out poet set, so this leakage risk is unresolved. If the human sample were replaced by poems from a disjoint set of poets, the measured detection difficulty, especially for style-matched D2/D3, could shift substantially, undermining the sweeping claim that current detectors cannot reliably detect LLM-generated modern Chinese poetry.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces AIGenPoetry, a benchmark for detecting whether modern Chinese poems were written by humans or generated by LLMs. The dataset contains 800 human poems by six professional poets and 41,600 poems generated by GPT-4.1, DeepSeek-V3, DeepSeek-R1, and GLM-4 under thirteen prompts covering intrinsic qualities (style, thought, theme), external structures (stanzas, lines), and emotions. The authors evaluate five zero-shot detectors plus a fine-tuned RoBERTa classifier in in-domain and generalization settings, and they report additional experiments across generation temperatures. The central claim is that current detectors cannot reliably detect LLM-generated modern Chinese poetry, with style-matched poems being the hardest and emotion-expressing poems—especially fear—the easiest to detect.","tokens_in":23822,"tokens_out":3308,"duration_ms":40327,"significance":"If the empirical findings hold, this is a useful first benchmark for an understudied domain. The dataset construction covers a broad prompt space, the human poems come from vetted professional poets, and the authors provide data/code links and a systematic comparison of six detectors. The observation that style-matched generation defeats most detectors, including near-chance performance for several zero-shot methods, is a concrete and falsifiable result. The temperature experiments also provide a practical insight for benchmark design. However, the strength of these contributions is currently limited by unresolved experimental-control issues, so the paper's sweeping conclusions outrun its evidence.","major_comments":[{"comment":"The generalization experiments are confounded by data leakage. All Di conditions reuse the same 800 human poems (Section 4: 'Human-written poems and LLMs-generated poems through Pi constitute a pair of data (HDi, LDi), denoted as Di'). In Table 4, training on D2+4+5 or D6−8 means the RoBERTa classifier has already seen the identical human poems that appear in the test conditions (D1, D9, D10−13, etc.). The human side of the test set can therefore be solved by memorization rather than by detecting LLM-generated text, inflating F1 and corrupting both the generalization conclusions and the relative difficulty ordering of poetry features. Please use a disjoint-poet split (e.g., train on a subset of poets, test on held-out poets) and report exact train/test splits for all experiments, including the in-domain set-ups.","section":"Section 4, Table 4"},{"comment":"No variance or significance information is reported. Every F1/AUROC is a single number with no seeds, standard deviations, or statistical tests. Some conclusions rest on small differences (e.g., Table 2 zero-shot F1 scores range 67.51–69.10), and RoBERTa fine-tuning is stochastic. Without multiple runs and error bars, the claims that one detector 'significantly outperforms' another, or that current detectors 'cannot be used as reliable tools,' are not supported. Please report means and standard deviations over at least five seeds, and use a paired test where appropriate.","section":"Tables 2–6"},{"comment":"For the zero-shot detectors, F1 depends on the decision threshold, but thresholds are not reported. AUROC is threshold-independent, yet the paper's central claims are framed primarily through F1. If thresholds were chosen on the test set, the F1 values are optimistic; if defaults were used, this should be stated and justified. Also, Table 2 shows Fast-DetectGPT with AUROC 43.24 on GPT-4.1, i.e., below chance, which suggests inverted scoring behavior; the paper should address why this occurs and how such detectors are being thresholded.","section":"Section 5, Tables 2–3"},{"comment":"The central claim is overbroad. The experiments cover four LLMs, one generation configuration (temperature 1.5, top_p 0.95), six poets, and a specific set of prompts. The conclusion that 'current detectors cannot be used as reliable tools to detect modern Chinese poems generated by LLMs' should be qualified to the tested conditions. Additionally, the style-difficulty conclusion is partly confounded: prompts P2 and P3 explicitly supply the human reference poem in the input, so the style condition differs from the baseline and structural conditions not only in 'style' but also in the presence of the reference poem. Please add a control condition or temper the causal language.","section":"Abstract, Section 6.2, Conclusion"}],"minor_comments":[{"comment":"Typo: 'top_p to0.95' should read 'top_p to 0.95.'","section":"Section 4"},{"comment":"The abbreviation 'Fast-Det.' is used in the table header but only defined in the caption as 'Fast-DetectGPT' in the first row; please define it in the header or caption consistently.","section":"Table 6"},{"comment":"The column headers 'P > Avg.' and 'P < Avg.' are unclear; define 'P' (presumably proportion of poems) in the caption.","section":"Table 8"},{"comment":"The text says 'the remaining five detectors have F1-scores clustered between 67.51% and 68.74%,' but Table 2 lists Fast-DetectGPT at 69.10 and Log-Likelihood at 68.00. Please correct the range or explain the discrepancy.","section":"Section 6.1"},{"comment":"The Limitations section addresses only classical/rhyming poetry. It should also mention the small number of poets, the single generation temperature, and the specific LLM set as limits on external validity.","section":"Limitations"},{"comment":"The ZHIPU reference is incomplete ('Zhipu ai devday glm-4' with no full title or venue); also the author name 'Fengying Y e' in the author block has an unintended space.","section":"References"}],"recommendation":"major_revision","confidential_remarks":"The paper's contribution is timely, but the editorial decision should hinge on whether the authors can fix the leakage and reporting issues. The validation logic (the benchmark is effective because detectors fail on it) is somewhat self-referential; it should be framed as a finding about current detectors rather than as a proof of benchmark quality. Also, the citation pattern is heavily self-referential (at least six citations to the authors' own prior work), which is worth monitoring but not disqualifying."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: this is a useful first benchmark for a niche but real task, and the style-imitation result is the one genuinely new empirical finding. But the paper oversells \"current detectors cannot be used as reliable tools\" when its own fine-tuned RoBERTa gets 91% in-domain, and the evaluation has a leakage problem in the generalization experiments that needs to be fixed before the claims are fully supported.\n\nWhat's good: the dataset is real and new (800 human poems, 41,600 LLM poems, 13 prompt types), and the prompt design is thoughtful—style, structure, emotion. The temperature sweep is a nice addition, and the authors correctly identify the GLM-4 length confound and show that structure constraints reduce detectability, which is evidence they're thinking about confounds. The limitations section is honest about scope.\n\nWhere it's soft: First, the headline claim is overbroad. Zero-shot detectors are indeed near chance, but RoBERTa reaches 91.17% in-domain and 84–89% out-of-domain. That's not \"cannot be used as reliable tools\"; it's \"these particular zero-shot methods should not be trusted.\" Second, the evaluation protocol is under-specified: no train/test split description, no thresholds for zero-shot F1, no error bars or seeds. Third—and this is the bigger issue—the generalization experiments reuse the same 800 human poems across training and test conditions. A model trained on D2+4+5 has seen the exact human poems that appear in the D1, D6-8, and D10-13 test sets. If upsampling of the human class is done before splitting, identical human texts can even end up in both train and test within a single condition. That means the RoBERTa OOD numbers and the relative difficulty ordering of style vs emotion could be inflated by human-poem memorization rather than by true detection of LLM text. The authors need to report a held-out poet set, or at minimum an explicit split that prevents human-text overlap, and ideally both. The in-domain style-hardest finding is less directly corrupted, but the shared human corpus still limits generalization claims.\n\nBottom line: the paper deserves peer review, and with a tightened protocol and a qualified headline it would be a solid benchmark. For now, treat the zero-shot negative result as interesting and the RoBERTa OOD numbers as not yet trustworthy.","headline":"Useful first benchmark with a real style-imitation finding, but the headline is overbroad and the evaluation needs a leakage fix before the out-of-domain numbers can be trusted.","tokens_in":24359,"tokens_out":4855,"would_cite":true,"duration_ms":55961,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that current AI-text detectors cannot reliably identify modern Chinese poems written by large language models, and that the hardest cases are poems imitating a human poet's style.","keywords":["AI-generated text detection","modern Chinese poetry","LLM poetry","benchmark dataset","style imitation","detector evaluation","temperature sensitivity"],"falsifier":"Give a detector only style-matched poem pairs (D2, D3) that are also matched for length and stripped of explicit emotion words, and require an F1 above 90 percent. If any current detector, or a simple style-based feature, reaches that level, the paper's conclusion that intrinsic qualities—especially style—are the hardest poetic features to detect would be contradicted.","tokens_in":23484,"feed_emoji":"🤖","tokens_out":7965,"duration_ms":73679,"temperature":0.7,"pith_summary":"This paper tries to show that existing detectors for AI-generated text cannot be trusted when the text is modern Chinese poetry, a genre that deliberately breaks grammar and convention. To test this, the authors built a dataset pairing 800 poems from six professional poets with 41,600 poems generated by four large language models under 13 different prompts that isolate style, thought, theme, stanza and line structure, and emotion. Running six detectors—five statistical and one fine-tuned RoBERTa classifier—they find the hardest cases are poems that imitate a human poet's style, where average F1 drops below 60 percent, while poems that literally name emotions like fear are the easiest to catch. If this is right, the practical consequence is that style-matching prompts, already common in real-world AI poetry generation, are an effective way to evade current detection tools.","feed_headline":"Style imitation defeats every AI-poetry detector tested","feed_subtitle":"A benchmark pairing 800 human poems with 41,600 LLM poems shows style-matched AI verse is the hardest to detect.","key_machinery":"The central object is the AIGenPoetry benchmark: a paired dataset of 800 human-written modern Chinese poems and 41,600 LLM-generated poems, produced by four LLMs under 13 prompts that isolate specific poetic dimensions. Prompts P2–P5 target intrinsic qualities (style imitation, thought and sentiment, theme); P6–P8 target external structure (stanza count, line count, both); P9–P13 target emotions (none, happiness, sadness, anger, fear). Each prompt yields a matched data pair (human vs. one LLM), so detector performance can be attributed to the specific feature being controlled. Six detectors—Fast-DetectGPT, LRR, Log-Likelihood, Log-Rank, Binoculars, and a fine-tuned RoBERTa classifier—are eva","core_discovery":"The paper's central claim is that current detectors cannot be used as reliable tools to detect modern Chinese poems generated by large language models. The evidence comes from a controlled benchmark in which four LLMs (GPT-4.1, DeepSeek-V3, DeepSeek-R1, GLM-4) each generated 800 poems per prompt across 13 prompt types, paired one-to-one with 800 human poems. In-domain results show average F1 around 72 percent on the baseline prompt and as low as 59.68 percent when poems are generated to imitate the style of a human poem (D2). GPT-4.1 style-matched poems are the hardest of all, with detectors averaging 48.73 percent F1. Poems that explicitly express a single emotion, especially fear, are the","pith_inferences":["The pattern suggests detectors are latching onto surface statistics—explicit emotion vocabulary, poem length, and stanza or line counts—rather than deeper stylistic or semantic signals; a detector built around imagery density, syntactic deviation, or cross-stanza coherence might close the style gap.","The one-to-one prompt pooling design could be stress-tested further by combining style imitation with other constraints (style plus structure, style plus emotion) to find which combinations are hardest, and by adding more human poets to reduce the risk that six poets' styles are unusually easy or hard to imitate.","The temperature results imply that a detector trained only on high-temperature, style-matched data would face the toughest possible distribution; this suggests a practical recipe for red-teaming future poetry detectors before deployment.","The benchmark's human poems come from six professional poets, so the finding is about matching those specific styles; how well detectors perform against a wider, more diverse human corpus remains an open question the paper does not resolve."],"forward_implications":["Style-matched generation is an effective evasion strategy: every detector tested loses at least 7 percentage points of F1 when the poem imitates a human poet's style, and four of the six lose more than 15 points.","Surface emotion words are a giveaway: poems that literally express emotions, especially fear, are the easiest to detect, which reflects a real asymmetry since human poets usually convey emotion implicitly.","Temperature matters: poems generated at temperature 1.5 are the hardest to detect, and detectors trained on 1.5 data generalize to lower-temperature data, so high-temperature generation is a reasonable worst-case benchmark setting.","Fine-tuned RoBERTa outperforms statistical detectors on this task, but its advantage shrinks on style-matched poems, meaning no current detector is reliable for real-world style-imitation use cases.","Detectors trained on intrinsic-quality prompts (style, thought, theme) generalize better to other feature types than detectors trained on structure or emotion prompts, pinpointing intrinsic qualities as the core unsolved difficulty."],"supporting_citations":[{"why":"provides the Fast-DetectGPT zero-shot detector evaluated on the benchmark.","marker":"(Bao et al., 2023)"},{"why":"provides the LRR zero-shot detection method evaluated on the benchmark.","marker":"(Su et al., 2023)"},{"why":"provides the Log-Likelihood scoring method evaluated as a detector.","marker":"(Solaiman et al., 2019)"},{"why":"provides the Log-Rank (GLTR) detection method evaluated as a detector.","marker":"(Gehrmann et al., 2019)"},{"why":"provides the Binoculars zero-shot detector evaluated on the benchmark.","marker":"(Hans et al., 2024)"},{"why":"provides the RoBERTa architecture fine-tuned as the sixth detector.","marker":"(Liu et al., 2019)"},{"why":"source of the GPT-4.1 model that generated 10,400 benchmark poems.","marker":"(Achiam et al., 2023)"},{"why":"source of the DeepSeek-V3 generator and of the temperature 1.5 / top_p 0.95 setting used for all generation.","marker":"(Liu et al., 2024)"},{"why":"source of the DeepSeek-R1 model that generated 10,400 benchmark poems.","marker":"(Guo et al., 2025)"},{"why":"source of the GLM-4 model that generated 10,400 benchmark poems.","marker":"(ZHIPU, 2024)"}],"fun_headline_variants":["AI poetry detectors fail on style-matched verse","Best AI-poetry detector falls to 49% F1 on GPT-4.1 style","New benchmark shows LLM poetry slips past all detectors","Style imitation makes AI poems undetectable to current tools","Poetry detectors beaten by AI style mimicry in new test"],"cache_read_input_tokens":2688,"weakest_assumption_plain":"The benchmark assumes that the 800 poems from six professional poets, and the 41,600 poems generated from them at temperature 1.5 with top_p 0.95, represent the real distribution of human-written and LLM-generated modern Chinese poetry; if the poet set, the prompt distribution, or the sampling settings changed, the measured detection difficulty could shift substantially.","fun_headline_variants_meta":{"raw":{"variants":["AI poetry detectors fail on style-matched verse","Best AI-poetry detector falls to 49% F1 on GPT-4.1 style","New benchmark shows LLM poetry slips past all detectors","Style imitation makes AI poems undetectable to current tools","Poetry detectors beaten by AI style mimicry in new test"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000599,"raw_usage":{"total_tokens":2637,"prompt_tokens":742,"completion_tokens":1895,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":486,"completion_tokens_details":{"reasoning_tokens":1809}},"tokens_in":486,"tokens_out":1895,"duration_ms":12409,"temperature":1.0,"reasoning_tokens":1809,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-05T12:20:49.834996+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Give a detector only style-matched poem pairs (D2, D3) that are also matched for length and stripped of explicit emotion words, and require an F1 above 90 percent. If any current detector, or a simple style-based feature, reaches that level, the paper's conclusion that intrinsic qualities—especially style—are the hardest poetic features to detect would be contradicted.","supporting_citations":[],"review_version":1}