{"id":"d693c443-c6f8-4af5-a41b-cdfc2363e87d","arxiv_id":"2608.10698","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"EVIL-Detect, a conflict-aware ensemble of edit-extent regression, zero-shot likelihood scoring, lexical statistics, and text rules, achieves 0.8888 macro-F1 and first place in NLPCC 2026 Shared Task 6 for Chinese three-class LLM-text detection.","lead":"A system called EVIL-Detect combines edit-strength regression, likelihood contrast, lexical statistics, and hand-written rules to tell apart human-written, LLM-generated, and LLM-refined Chinese text. It reports first place in the NLPCC 2026 Shared Task 6 with a macro-F1 of 0.8888 on the official test set.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Timeline ambiguity: ablation on 'released testp2 labels' may mean fusion thresholds were tuned on the hidden test set, invalidating the reported 0.8888 macro-F1 and top-rank claim.","rationale":"The reader's weakest assumption concerns the EditLens distance proxy d(h,t). That is a plausible scientific risk, but the paper's own component analysis on testp1 shows EditLens as a standalone module reaches 0.8494 macro-F1, indicating the proxy is empirically useful for this benchmark even without perfect human alignment. The central claim of this shared-task paper is specifically the official top rank and macro-F1 of 0.8888 on the hidden test. The most load-bearing condition for that claim is the integrity of the evaluation. The wording in Sec. 4.4, 'released testp2 labels,' creates a timeline ambiguity: if those labels were accessible during development, the fusion thresholds could have been selected on the test set, making the reported testp2 result circular. The paper should state, with a date, that testp2 labels were released only after the final system was fixed. This is a concrete, checkable condition. I therefore keep the reader's CONDITIONAL verdict, but shift the emphasis from the EditLens proxy to the evaluation-timeline issue. Agreement with reader is partial because the reader separately flagged this ambiguity in the rationale but did not make it the primary weakest assumption.","tokens_in":9077,"tokens_out":9670,"duration_ms":96999,"concrete_test":"Check the official NLPCC 2026 Shared Task 6 repository (https://github.com/NLP2CT/NLPCC-2026-Task6-Detection) and the task announcement for the release date of testp2 labels and the submission deadline. If testp2 labels were released before the authors froze their fusion configuration, the reported testp2 performance is suspect. If they were released only after the official evaluation, the concern is resolved and the paper's claim stands.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 4.4 states: 'We conduct an ablation of the fusion module on the released testp2 labels.' The official task description (ref [14]) designates testp2 as a hidden test phase. The paper never states when testp2 labels were released relative to the finalization of the 'validation-calibrated' fusion thresholds introduced in Sec. 3.6 (the two EditLens boundary settings τ(k) and the resolver thresholds γH, βT, γHL, γTL). If testp2 labels were available before the system was frozen, the reported testp2 macro-F1 of 0.8888 is a test-set-tuned number and the central claim of ranking first on a hidden out-of-distribution evaluation is not an independent result. Even if the ablation was purely post-hoc, the ambiguity prevents verification. This is the most load-bearing concern because the headline metric and rank are the paper's main contribution.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents EVIL-Detect, a multi-signal ensemble system for NLPCC 2026 Shared Task 6, which requires three-class classification of Chinese text into human-written text (HWT), LLM-generated text (LGT), and LLM-refined text (HLT). The system combines an EditLens-inspired edit-extent regression model, a semantic variant called Soft-EditLens, a zero-shot EchoPrompt likelihood-contrast module, lexical frequency statistics, and conservative text rules, all integrated through calibrated decision boundaries and a conflict-aware fusion module. The authors report macro-F1 scores of 0.8913 on testp1 and 0.8888 on testp2, and claim first place in the official evaluation. The paper includes component analysis on testp1 and an ablation study on testp2.","tokens_in":9406,"tokens_out":6228,"duration_ms":65595,"significance":"If the reported results are valid and independent, EVIL-Detect would be a strong published system for Chinese three-class LLM-text detection, with a useful design: using edit-extent regression as the base signal, treating likelihood-based and lexical methods as auxiliary vote sources, and resolving boundary conflicts with a deterministic rule set. The paper's strengths include a released code repository, official leaderboard scores, and explicit component and ablation tables. However, the significance is contingent on two unresolved points: the chronological relationship between the release of testp2 labels and the tuning of fusion thresholds, and the reproducibility of the 'calibrated' thresholds, whose values are not reported. The testp2 ablation, in particular, casts doubt on whether the headline 0.8888 macro-F1 is an independent evaluation or a test-set-tuned number.","major_comments":[{"comment":"The ablation is said to be conducted 'on the released testp2 labels,' but §4.1 describes testp2 as a hidden out-of-distribution test phase. The paper never states whether these labels were available before the fusion thresholds in §3.6 (τ^(k)_1, τ^(k)_2, γ_H, β_T, γ_HL, γ_TL) were finalized. If they were, the reported 0.8888 macro-F1 and the first-place claim are test-set-tuned and not an independent evaluation; if they were not, the text should say so explicitly and should report the validation-tuned configuration's score on testp2 separately. Please clarify the chronological ordering and, if applicable, replace the testp2-based ablation with one on a held-out validation set.","section":"§4.4, Table 7"},{"comment":"The fusion logic depends on four thresholds and two EditLens boundary settings, but their numerical values and the calibration procedure are not given. The paper says 'validation-calibrated' without defining the validation set, the calibration objective, or whether the calibration was performed once before the test phases. Without this information, the claimed 'calibrated decision boundaries' cannot be reproduced or checked. Provide the exact threshold values, the validation set construction, and the calibration criterion.","section":"§3.6"},{"comment":"The method assumes that an edit-extent score d(h,t) defined using the paired human source h can be predicted from the target text x alone at inference time. The paper reports downstream macro-F1 but no direct evidence that the predicted scores are calibrated or that the HLT distribution is separable from the two anchors. Please report the correlation between predicted and true edit-extent on a held-out split, and either a calibration curve or histograms of predicted scores by true class, to support the claim that the EditLens score, which anchors the entire fusion in Eq. (5), is reliable.","section":"§3.3, Eqs. (1)-(3)"},{"comment":"All macro-F1 numbers are point estimates from single runs, and testp2 has only 1,152 samples. The reported differences, such as the 4.19-point improvement over EditLens on testp1 and the 0.8411-to-0.8888 improvement in the ablation, may be within sampling noise. Provide bootstrap confidence intervals or a significance test for the main comparisons, especially those that support the robustness claim.","section":"Tables 4, 5, 7"}],"minor_comments":[{"comment":"The variable z_soft is introduced in the resolver description but is not formally defined; define it as the pair (z_r, z_b) or as a named tuple before presenting the conflict-resolution rules.","section":"§3.6, Eq. (6)"},{"comment":"The nine binary LGT-support votes are summed with equal weight in v(x); state explicitly whether any confidence calibration or per-vote weighting is applied, or whether the votes are intentionally unweighted.","section":"Table 3"},{"comment":"The sentence 'The small gap between the two phases indicates that the multi-signal design remains stable' is stronger than the evidence allows, since there are only two test phases of different sizes; rephrase as 'suggests' or support it with per-phase confidence intervals.","section":"§4.2"},{"comment":"Calling the testp2 analysis an 'ablation' is appropriate only if all configurations were fixed before the testp2 labels were released; otherwise, label it a post-hoc analysis to avoid implying that testp2 was an untouched held-out set.","section":"§4.4"},{"comment":"Clarify how Eqs. (2) and (3) are combined with the r(h)=0, r(g)=1, r(t)=d(h,t) assignment; in particular, state which distance definition is used for the EditLens rank044 target and how the Soft-EditLens phrasal distance is aligned with the same HWT-to-LGT axis.","section":"§3.3, Eqs. (1)-(3)"},{"comment":"The zero-shot module uses both Qwen3.5-4B and Qwen2.5-1.5B instruct/base model pairs, but the table does not specify which checkpoints are instruct models and which are base models; add this information for reproducibility.","section":"Table 2"}],"recommendation":"major_revision","confidential_remarks":"The central risk is the testp2 timing issue: if the authors cannot demonstrate that the fusion thresholds and the two EditLens boundary settings were frozen before the testp2 labels were available, the headline result is not an independent evaluation. I recommend asking the editors to require a clear timeline and, if necessary, a re-analysis on a truly held-out validation set. The self-citation of the EchoPrompt preprint [25] should also be checked for overlap with the present paper's contribution, though this is not a blocker if the module is properly attributed."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Bottom line: this is a well-structured shared-task system report with a plausible first-place result, but the headline 0.8888 macro-F1 is not independently verifiable and a genuine timeline ambiguity around the testp2 ablation needs to be cleared before I'd trust it as an honest out-of-distribution score.\n\nWhat is actually new: the ensemble is the contribution. EditLens, EchoPrompt, lexical statistics, and text rules are all prior work, but the editing-extent continuum target (HWT=0, LGT=1, HLT=d(h,t)) and the conflict-aware fusion with validation-calibrated thresholds are a legitimate extension for Chinese three-class OOD detection. Credit goes to the honest component analysis: Table 5 shows most modules are weak standalone, which justifies the multi-signal design rather than hiding behind a single number. The testp2 ablation, if truly post-hoc, shows a real improvement on boundary classes, with LGT→HLT errors cut roughly in half.\n\nSoft spots, in order of severity. First, Section 4.4 says the ablation was run on \"released testp2 labels,\" but the paper never states when those labels became available relative to freezing the fusion thresholds. If thresholds were fitted after seeing testp2, the 0.8888 is a test-set-tuned number and the first-place claim loses its independent value. The paper uses the word \"released,\" which suggests the authors had them, but for how long is unclear. Even if the ablation was purely post-hoc, the ambiguity is trivially resolvable and should be stated explicitly. Second, no error bars, exact threshold values are omitted, and the code link is unversioned, which limits reproducibility. Third, the EditLens distance proxy d(h,t) is validation-fitted and assumes source-free prediction at inference; if the proxy doesn't track human judgments of editing intensity, the entire continuum signal is miscalibrated. That risk is inherent to the design, and the paper only checks it via the rank044 separability sweep, not against human or oracle editing scores.\n\nWho this is for: researchers working on Chinese LLM-text detection, especially three-class and OOD settings, and shared-task participants comparing system architectures. It is a system report, not a conceptual breakthrough.\n\nRecommendation: this deserves peer review, conditionally. A serious referee should ask for the exact timeline of testp2 label release, the calibration threshold values, and a versioned code release. Without that, the central claim stays conditional.","headline":"Competent shared-task system report with a plausible top result, but the testp2 ablation timeline ambiguity undercuts the headline until clarified.","tokens_in":9808,"tokens_out":3599,"would_cite":false,"duration_ms":37943,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Treating LLM-refined text as a continuous editing extent, plus conflict-aware fusion of four signal types, achieves 0.8888 macro-F1 and first place in NLPCC 2026 Shared Task 6.","keywords":["LLM-generated text detection","three-class classification","edit-extent regression","conflict-aware fusion","zero-shot detection","Chinese text","out-of-distribution robustness","ensemble learning"],"falsifier":"Take a sample of held-out (HWT, HLT) pairs from the evaluation distribution, have human annotators rate the degree of editing on a continuous scale, and if the Spearman correlation between those ratings and the EditLens score $s_E(x)$ predicted from the refined text alone is close to zero, the calibrated boundaries in Eq. (5) are not anchored to real editing extent and the fusion is built on a miscalibrated signal.","tokens_in":8860,"feed_emoji":"🤖","tokens_out":12047,"duration_ms":98917,"temperature":0.7,"pith_summary":"EVIL-Detect is a system for three-class Chinese text detection, sorting text into human-written (HWT), LLM-generated (LGT), and LLM-refined (HLT) classes. The paper's core move is to treat HLT not as a hard third class but as an intermediate editing state on a continuous axis, with a regression model trained to predict an editing-extent score between a human source and its refined version. At test time the source is not needed: the model predicts the score from the text alone. Around that backbone, the system layers zero-shot likelihood-contrast votes, lexical log-odds statistics, and conservative text rules, then fuses them with two calibrated decision boundaries and a conflict-aware resolver. The paper reports this design is stable under out-of-distribution shifts, achieving macro-F1 of 0.8913 on one hidden test phase and 0.8888 on the other, ranking first in the official evaluation.","feed_headline":"Three-way AI-text detector hits 0.8888 macro-F1 and wins NLPCC 2026.","feed_subtitle":"It treats LLM-refined text as an editing degree and fuses several signals, staying accurate on out-of-distribution data.","key_machinery":"The central object is the editing-extent score $s_E(x)$ produced by an EditLens regression model: a scalar in $[0,1]$ trained to predict $d(h,t)$, the distance between a human source text $h$ and its LLM-refined version $t$, with $r(h)=0$ and $r(g)=1$ as anchors. The distance is computed either by weighted character n-gram cosine distance (Eq. 2) or by semantic phrase-level matching (Eq. 3). This score is discretized with two calibrated boundary pairs $(\\tau^{(k)}_1, \\tau^{(k)}_2)$ into base labels (Eq. 5); the fusion module keeps the label when the two discretizations agree, and otherwise applies a conflict-aware resolver $R(\\cdot)$ that uses Soft-EditLens scores and nine binary LGT-support votes from zero-shot likelihood contrast and lexical log-odds statistics, followed by conservative text rules that flag HTML/XML markup as LGT and rewriting traces as HLT. The machinery's role is to turn an inherently ordinal phenomenon — how much an LLM has edited human text — into a regression target that can be learned without paired sources at test time, and then into deterministic decisions with calibrated thresholds.","core_discovery":"The central discovery is that a three-way HWT/LGT/HLT detector can be made reliable by assigning HLT a soft, instance-dependent editing extent $r(t) = d(h,t)$ anchored at $r(h)=0$ for human text and $r(g)=1$ for fully generated text, rather than treating HLT as a third independent class. The paper instantiates this with two edit-extent regressors — EditLens, using a sweep-selected weighted character n-gram distance (Eq. 2), and Soft-EditLens, using semantic phrase matching and an ordinal bucket head (Eq. 3) — and shows that the best standalone EditLens configuration reaches 0.8494 macro-F1 on the first test phase. The full EVIL-Detect fuses two calibrated discretizations of the EditLens score (Eq. 5) through a conflict-aware resolver that consults Soft-EditLens regression/bucket outputs and a panel of nine binary LGT-support votes from EchoPrompt and lexical statistics, then applies high-precision text rules. On the official test phases the system reports macro-F1 0.8913 (testp1) and 0.8888 (testp2), with HLT remaining the hardest class (F1 around 0.84); the ablation attributes most of the gain over EditLens-only (0.8411) to conflict-aware integration plus rules (0.8816) and the second calibrated boundary (0.8888).","pith_inferences":["If the editing-extent framing transfers, similar soft-label or ordinal targets could improve binary HWT-vs-AI detection in other languages, since the intermediate class is the same phenomenon everywhere; the paper only demonstrates Chinese news and academic text.","The conflict-aware fusion rules are manually engineered with validation-calibrated thresholds; an implicit testable extension is whether a learned meta-classifier over the same signals would match or beat the hand-written resolver, especially on unseen boundary distributions.","Because the text rules treat raw HTML/XML markup as near-certain LGT, the system would likely misclassify genuinely human-authored markup-heavy content, such as web pages or code documentation, if such content appears in future test sets."],"forward_implications":["For Chinese three-class detection, an edit-extent regression target is a stronger base signal than direct generative classification: direct QLoRA/SFT variants scored 0.1690–0.1896 macro-F1 while EditLens alone scored 0.8494.","The largest remaining error concentration is at the HWT/HLT and HLT/LGT boundaries; conflict-aware fusion cut LGT→HLT errors from 59 to 30 and HLT→LGT errors from 58 to 28 on testp2.","The small gap between testp1 (0.8913) and testp2 (0.8888) indicates that the design transfers across the two out-of-distribution evaluation splits.","Because each auxiliary module (EchoPrompt, lexical statistics) is weak as a standalone three-class predictor (0.63 and 0.55 macro-F1), the system's performance comes from using them as evidence votes in fusion, not as independent classifiers."],"supporting_citations":[{"why":"Supplies the edit-extent regression formulation that the paper adopts as its backbone, replacing discrete classification with continuous editing-degree prediction.","marker":"[15]"},{"why":"The official NLPCC 2026 Shared Task 6 description that defines the three-class HWT/LGT/HLT benchmark and the macro-F1 evaluation metric.","marker":"[14]"},{"why":"Provides the EchoPrompt zero-shot likelihood-contrast module that generates the LGT-support votes used in fusion.","marker":"[25]"},{"why":"The CUDRT dataset from which the official training triplets (h, g, t) are sampled and adapted.","marker":"[12]"},{"why":"The DetectRL-X Chinese split that supplies the out-of-distribution test phases (testp1 and testp2).","marker":"[13]"},{"why":"Binoculars is the representative training-free baseline compared in the alternative-design analysis, standing in for likelihood-based detectors the system must beat.","marker":"[10]"},{"why":"LoRA is the parameter-efficient fine-tuning method used for both EditLens and Soft-EditLens regression models.","marker":"[22]"}],"fun_headline_variants":["EVIL-Detect wins NLPCC 2026 with 0.8888 macro-F1","Edit-extent fusion wins AI-text detection at NLPCC 2026","Three-way text detector scores 0.8888, takes first at NLPCC","How to spot AI text: editing degree approach ranks first"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The whole system rests on the premise that the degree to which an LLM has edited a human text can be captured by a scalar distance between the human source and the refined text, and that a model can learn to predict that distance from the text alone, without ever seeing the source at test time.","fun_headline_variants_meta":{"raw":{"variants":["EVIL-Detect wins NLPCC 2026 with 0.8888 macro-F1","Edit-extent fusion wins AI-text detection at NLPCC 2026","Three-way text detector scores 0.8888, takes first at NLPCC","How to spot AI text: editing degree approach ranks first"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000742,"raw_usage":{"total_tokens":3344,"prompt_tokens":1013,"completion_tokens":2331,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":629,"completion_tokens_details":{"reasoning_tokens":2248}},"tokens_in":629,"tokens_out":2331,"duration_ms":18833,"temperature":1.0,"reasoning_tokens":2248,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T19:06:39.883036+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a sample of held-out (HWT, HLT) pairs from the evaluation distribution, have human annotators rate the degree of editing on a continuous scale, and if the Spearman correlation between those ratings and the EditLens score $s_E(x)$ predicted from the refined text alone is close to zero, the calibrated boundaries in Eq. (5) are not anchored to real editing extent and the fusion is built on a miscalibrated signal.","supporting_citations":[{"cited_title":"https://github.com/NLP2CT/ NLPCC-2026-Task6-Detection (2026)","cited_arxiv_id":null,"evidence_quote":"The official NLPCC 2026 Shared Task 6 description that defines the three-class HWT/LGT/HLT benchmark and the macro-F1 evaluation metric."},{"cited_title":"Once a Response, Always a Response: Detecting LLM-generated Text via Latent Prompt Restoration","cited_arxiv_id":"2608.05741","evidence_quote":"Provides the EchoPrompt zero-shot likelihood-contrast module that generates the LGT-support votes used in fusion."},{"cited_title":"ACM Transactions on Intelligent Systems and Technology17(2), 1–35 (2026)","cited_arxiv_id":null,"evidence_quote":"The CUDRT dataset from which the official training triplets (h, g, t) are sampled and adapted."},{"cited_title":"In: Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp","cited_arxiv_id":null,"evidence_quote":"The DetectRL-X Chinese split that supplies the out-of-distribution test phases (testp1 and testp2)."},{"cited_title":"In: Proceedings of the 41st International Conference on Machine Learning, pp","cited_arxiv_id":null,"evidence_quote":"Binoculars is the representative training-free baseline compared in the alternative-design analysis, standing in for likelihood-based detectors the system must beat."}],"review_version":1}