{"id":"830186a7-f53a-4c96-8bd7-4016010b48fa","arxiv_id":"2507.05157","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"Fine-tuned GPT-4o-mini detects AI versus human text at 95.5% F1 on the DeFactify test set, but identifying the specific generator LLM reaches only 47% F1 with BERT.","lead":"The authors fine-tuned GPT-4o-mini, LLaMA-3 8B, and BERT to tell human-written text from AI-generated text, and to identify which AI model wrote a given passage. GPT-4o-mini reached 95.5% F1 on the binary task, while identifying the exact generator stayed hard at 47% F1 with BERT.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The reported 0.9547 Task-A F1 is an upper bound if the ~200 Azure-filtered test samples were excluded; the paper never states whether they were dropped or scored as errors, so the headline claim is not verified on the full test set.","rationale":"The reader's weakest assumption identifies the same load-bearing concern: the ambiguous treatment of ~200 content-filtered test samples. My reading of Section 4 confirms the ambiguity: the paper mentions the error and states a 2% effect, but provides no corrected F1 and no explicit denominator. Because the abstract presents 0.9547 as the achieved accuracy/F1, this is a correctness risk for the paper's central claim. I considered other concerns: the abstract uses 'accuracy' for what Table 6 labels 'F1', the validation F1 of 1.00 for BERT is suspiciously high, and no code is released. These are secondary: the mislabel is a reporting issue, the high validation F1 could be overfitting but the test F1 is what is claimed, and code absence hampers verification but is not itself a flaw. The filtering issue is the most direct threat to the quantitative claim. I recommend keeping the reader's CONDITIONAL verdict, since the concern is real and requires author clarification; it does not by itself invalidate the qualitative finding that simple instruction tuning gives strong Task-A separation, but it does mean the exact number should not be accepted at face value. The proposed test is concrete and can be done by the authors or organizers.","tokens_in":5801,"tokens_out":4837,"duration_ms":53523,"concrete_test":"Obtain the per-sample predictions and the exact set of the ~200 filtered test items from the authors or the DeFactify organizers (or re-run the Azure API on the test set if access and cost allow). Recompute Task-A macro F1 two ways: (1) excluding filtered samples exactly as the paper likely did, reproducing 0.9547; (2) scoring every filtered sample as a misclassification (e.g., predicted wrong label). If the F1 in (2) drops by more than the stated 2% or falls below BERT's 0.7670, the abstract should be corrected to report the F1 on the full test set with the filtered samples counted, and the current 0.9547 should be labeled as an upper bound.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central quantitative claim, GPT_4o-mini reaching 0.9547 macro F1 on Task-A, rests on an unspecified handling of test samples that triggered Azure content filtering. Section 4 states that approximately 200 test samples produced no model response due to the Azure content management policy and that 'this has limited the GPT_4o-mini model performance by 2%,' but it never says whether these samples were excluded from the denominator or counted as errors. If they were excluded, 0.9547 is a conditional F1 on the ~10,763 responsive samples, not an unbiased estimate on the full 10,963-sample test set. The bias is unlikely to be random: Section 2.1 notes the dataset includes medical and election topics, and content filters are known to trigger on such topics, so the missing samples may be systematically drawn from particular domains or styles. The paper also labels the result 'accuracy' in the abstract while Table 6 reports F1, adding ambiguity. This is load-bearing because the headline claim is a precise number; an undefined denominator makes the number non-reproducible. The Task-B BERT result (0.4698) is not affected by this particular filtering issue, but the paper's strongest quantitative assertion is the Task-A number.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This manuscript reports experiments on the DeFactify 2025 Text Counter Turing Test benchmark for two tasks: Task-A, binary classification of human-written versus machine-generated text, and Task-B, identification of the specific LLM (among six LLMs plus human) that produced a given text. The authors fine-tune GPT-4o-mini (Task-A only), LLaMA-3 8B (Task-A and Task-B), and BERT (Task-A and Task-B) using simple instruction prompts. The abstract claims accuracies of 0.9547 on Task-A for GPT-4o-mini and 0.4698 on Task-B for BERT. Validation results show very high F1 scores (up to 1.00 for BERT on Task-A), while test results show large drops for BERT and LLaMA-3 8B on both tasks. The paper concludes that simple prompts are effective for Task-A but that Task-B requires more sophisticated approaches.","tokens_in":6070,"tokens_out":12401,"duration_ms":110157,"significance":"The empirical result that a fine-tuned GPT-4o-mini reaches 0.9547 macro F1 on Task-A of an external shared benchmark is a potentially useful data point for the AI-text-detection community, and the benchmark choice avoids circularity in the evaluation. The paper does not introduce a new method, but it provides a head-to-head comparison of three model families under fine-tuning on the same data, with hyperparameters tabulated. However, the contribution is limited by the lack of code release, the absence of confidence intervals, and the unspecified handling of filtered test samples. The Task-B results (0.4698 for BERT, 0.14 for LLaMA-3 8B) are modest, and the paper's own analysis is thin.","major_comments":[{"comment":"The paper reports that 'Open AI's GPT_4o-mini fine-tuned model did not provide answers for approximately 200 test samples' due to Azure content filtering, and that 'This has limited the GPT_4o-mini model performance by 2%,' but it never states whether these samples were excluded from the denominator of the reported Task-A F1 score of 0.9547 or counted as errors. If they were excluded, the headline F1 is conditional on the responsive subset and is potentially biased upward, especially because Section 2.1 notes the dataset includes medical and election topics that are plausible triggers for content filtering. The authors must specify the exact scoring protocol, report the number of valid samples used in the denominator, and if possible provide the F1 computed with the filtered samples scored as errors or re-run the evaluation with the filter disabled.","section":"Section 4, Table 6"},{"comment":"The abstract states that 'the fine-tuned version of GPT_4o-mini and the BERT model has achieved accuracies of 0.9547 for Task-A and 0.4698 for Task-B,' but Section 4 explicitly says performance is measured using macro-averaged precision, recall and F1, and Table 6 columns are headed 'Task-A F1' and 'Task-B F1.' The reported values are therefore macro F1 scores, not accuracies. This mislabeling in the abstract should be corrected (e.g., change 'accuracies' to 'macro F1 scores') to avoid misleading readers about the nature of the headline result.","section":"Abstract, Section 4, Table 6"},{"comment":"The validation-to-test F1 drops are dramatic: BERT falls from 1.00 to 0.7670 on Task-A and from 0.98 to 0.4698 on Task-B, while LLaMA-3 8B falls from 0.93 to 0.14 on Task-B, which is at the chance level for a 7-class problem. The paper merely states that the models 'have not generalized well on the test dataset' without investigating possible causes such as overfitting, label leakage, or distribution shift. In particular, a validation F1 of exactly 1.00 for BERT on Task-A is a red flag that the validation set may not be representative of the test distribution; the authors should analyze this discrepancy and discuss its implications for the reliability of the reported test numbers.","section":"Tables 4-6"},{"comment":"The fine-tuning configuration for GPT-4o-mini is reported as 'Batch Size: 2 to 4; Epochs: 1 to 2,' which is a range rather than a single configuration. The paper does not state the exact values used for the final model, nor the number of training examples used, nor the model-selection criterion on the validation set. Without these details, the reported 0.9547 is not independently reproducible. Please provide the exact settings and selection procedure.","section":"Section 3.2, Table 3"}],"minor_comments":[{"comment":"The sentence 'The labels in train and validation set are equally distributed as shown in the Table 1' is contradicted by Table 1, which shows 7,321 human versus 43,926 machine samples in the training set and 1,569 versus 9,414 in the validation set. Please correct the statement to reflect the actual imbalanced distribution.","section":"Section 2.1"},{"comment":"The text says 'The training and validation losses for Tasks-A and B are plotted in Figure 2a and Figure 3, respectively,' but Figure 3 is an illustration of a filtered input sample, and Figure 2b contains the Task-B loss curves. The reference should be to Figure 2a and Figure 2b.","section":"Section 3.2"},{"comment":"Reference [3] contains the placeholder 'Accessed: (Use the date of access).' Please provide the full access date.","section":"References"},{"comment":"Model naming is inconsistent: 'GPT_4o-mini' appears in the abstract and most sections, while Table 6 uses 'GPT_4-o.' Please standardize to one name (e.g., 'GPT-4o-mini').","section":"Throughout"},{"comment":"The sentence 'For Task-A, we fine-tuned OpenAI's GPT_4o-mini and Google's BERT' omits LLaMA-3 8B, which is listed for Task-A in Table 2 and reported in Table 4. Please correct the list of models.","section":"Section 3"},{"comment":"The statement 'The final results shown in the Table 6 are obtained by combining GPT_4O-mini Task-A results with BERT model Task-B results' is ambiguous because the table also reports BERT Task-A and LLaMA-3 8B Task-B results. Clarify which rows constitute the final combined system.","section":"Section 3.2, Table 6"},{"comment":"The paper reports no confidence intervals or significance tests for any F1 scores; in particular, the claim that GPT-4o-mini outperforms BERT on Task-A (0.9547 vs. 0.7670) would benefit from a statistical test or at least a statement about variance.","section":"Tables 4-6"},{"comment":"The author footnote contains the stray string 'envel⌢pe-⌢penc' before the email address; this appears to be a formatting artifact that should be removed.","section":"Author footnote"}],"recommendation":"major_revision","confidential_remarks":"This is a short empirical paper whose main value is a benchmark result; the novelty is limited. The major issue is the unspecified handling of the Azure-filtered test samples, which the authors can fix by clarifying the denominator and, ideally, re-running the evaluation. The abstract's 'accuracies' mislabel should also be corrected. If these are addressed, the paper could be acceptable for a workshop venue; I recommend major revision for now."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: this is a compact DeFactify shared-task writeup. The one new empirical thing is that instruction fine-tuning a GPT-4o-mini with a simple prompt reaches 0.9547 macro F1 on Task-A (human vs. machine), and BERT gets 0.4698 on Task-B (which generator). That result is plausible, but it is not fully reproducible from the paper because the treatment of roughly 200 Azure-filtered test samples is never stated.\n\nWhat it does well: it uses the official DeFactify 2025 benchmark, gives validation and test numbers for three model families, and is transparent about the validation-to-test drop. The instruction fine-tuning setup is simple and clearly described enough to replicate, modulo the filtering issue. It also correctly concludes that attribution (Task-B) is much harder than binary detection, which matches prior work.\n\nSoft spots: the biggest is the missing denominator detail. Section 4 reports that about 200 test samples produced no answer because Azure content filtering triggered, and says this limited performance by 2%, but it does not say whether those samples were dropped from the F1 calculation or scored as errors. If they were dropped, 0.9547 is a conditional F1 on the responsive subset, and with medical and election content in the data, the bias may not be random. The abstract also calls F1 \"accuracy,\" which muddies the headline. On top of that, there are no confidence intervals or significance tests, and the gap between validation (100% for BERT) and test (76%) is large enough to raise overfitting concerns, though the authors do acknowledge the generalization problem. No code is released, so independent verification requires reimplementation.\n\nWho this is for: someone working on AI-text detection who wants a reference point for what simple fine-tuning gets on DeFactify, or someone preparing a shared-task system description themselves. It is not a methods paper.\n\nRecommendation: I would send it to peer review, because the benchmark result is concrete and the main flaw is fixable in revision. A referee should insist on a precise statement of how the filtered samples were handled, corrected terminology, and ideally error bars or at least a breakdown. As is, treat the 0.9547 as an upper bound rather than the final word.","headline":"A concrete but under-specified DeFactify shared-task result: simple instruction tuning of GPT-4o-mini reaches 95.5% macro F1 on binary detection, yet the paper never states how ~200 Azure-filtered test samples were handled, so the headline number is not fully verified.","tokens_in":6612,"tokens_out":3677,"would_cite":false,"duration_ms":44171,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Fine-tuning GPT-4o-mini with a single instruction prompt distinguishes human from machine text at 0.9547 macro F1 on the DeFactify unseen test set, while the best model for identifying the specific generator, BERT, reaches only 0.4698.","keywords":["AI-generated text detection","instruction fine-tuning","GPT-4o-mini","BERT","LLaMA-3 8B","human vs machine text classification","generator attribution","DeFactify benchmark"],"falsifier":"Re-run the Task-A evaluation on the same test set with two rules for the ~200 content-filtered samples: count all of them as errors, and count all of them according to their true labels; if the macro F1 drops by more than a couple of points under the error rule, the claim of 0.9547 is not robust to the filtering artifact. The same recalculation should be repeated in a deployment setting with no content filtering applied at all.","tokens_in":5606,"feed_emoji":"🤖","tokens_out":8982,"duration_ms":88168,"temperature":0.7,"pith_summary":"This paper tests whether simple instruction fine-tuning can separate human-written from machine-generated text (Task-A) and identify which specific LLM generated a text (Task-B). On the DeFactify benchmark, fine-tuning OpenAI's GPT-4o-mini with a one-sentence prompt reaches a macro F1 of 0.9547 on the unseen test set for Task-A, while BERT, fine-tuned the same way, reaches 0.4698 for Task-B. The authors conclude that binary detection is largely achievable with simple prompts, but identifying the exact generator remains an open problem requiring larger models and richer instructions. If correct, this offers a low-effort recipe for building human-vs-machine text detectors and redirects attention to the harder attribution task.","feed_headline":"One-line prompt turns GPT-4o-mini into a 95% AI-text detector","feed_subtitle":"Naming the exact AI writer is far harder: that task tops out at 47% macro F1 with BERT.","key_machinery":"The mechanism is supervised fine-tuning from instruction-formatted examples: for each input text, a short classifier prompt such as 'classify whether the given text is written by human or machine' is prepended, and the target label is used as the supervised response. GPT-4o-mini and LLaMA-3 8B are adapted by continuing language-model training on these prompt-label pairs, while BERT is fine-tuned as a text classifier with the same labels. The central objects are the instruction-tuned dataset, the three fine-tuned models, and the macro F1 evaluation used by the benchmark. BERT's role is notable: despite being older and smaller than the generative LLMs, it gives the best attribution result (0.4698 Task-B F1) and a strong binary result (0.7670 Task-A F1), because it can be trained directly as a multi-class classifier.","core_discovery":"The paper's central claim is that instruction fine-tuning with simple prompts is enough to make a strong binary detector. After prepending a one-sentence classifier prompt to each text and fine-tuning on labeled examples, GPT-4o-mini reaches 0.9547 macro F1 in distinguishing human-written from machine-generated text on the DeFactify unseen test set. BERT, fine-tuned for the seven-way attribution task, reaches 0.4698 macro F1 in identifying which of six LLMs generated a text. LLaMA-3 8B, fine-tuned with 4-bit LoRA, achieves only 0.14 macro F1 on the same attribution task. The authors also report that all models' test performance falls well below their validation performance, with BERT dropping from near-perfect validation scores to 0.7670 (Task-A) and 0.4698 (Task-B) on the test set, and they conclude that pinpointing the generator will require larger models, longer contexts, and detailed prompting instructions.","pith_inferences":["The paper does not state whether the roughly 200 test samples blocked by the provider's content filter were excluded from the F1 denominator; a reader deploying this detector in an environment without such filtering should treat 0.9547 as a ceiling rather than the expected operational accuracy.","The large validation-to-test gap suggests the benchmark's train and test distributions differ substantially, so a user checking only validation scores would overestimate real-world performance by a wide margin on Task-B.","Because the winning recipe is just a prepended instruction and standard fine-tuning, the same approach could be tested cheaply for other binary provenance questions, such as detecting machine-translated or AI-summarized text, an experiment the paper does not run.","The paper's own data analysis shows human text has wider lexical diversity than machine text; adding lexical-diversity and sentence-length features as auxiliary signals could be a testable way to improve Task-B, where simple prompts underperform."],"forward_implications":["A detector for human-vs-machine text can be built by fine-tuning a commercial LLM with a one-sentence instruction prompt, reaching 95.47% macro F1 on the benchmark's unseen test set.","Identifying the exact generator remains largely unsolved: the best attribution score is 46.98% macro F1 (BERT), and LLaMA-3 8B reaches only 14%, far below a usable level.","Validation performance is much higher than test performance (BERT reaches 100% validation F1 for Task-A versus 76.7% on test), so models fine-tuned this way can overfit to the validation distribution.","Simple prompts suffice for binary detection, but the paper concludes that attribution will require larger models, longer contexts, and prompts that describe each generator's linguistic characteristics.","On the test set, combining GPT-4o-mini's Task-A result with BERT's Task-B result yields the paper's headline pair of 0.9547 and 0.4698; neither model alone solves both tasks."],"supporting_citations":[{"why":"Supplies the labeled train/validation/test texts for both tasks; the paper's reported F1 scores are measured on this dataset.","marker":"[12]"},{"why":"Defines the DeFactify task setup, labels, and macro F1 evaluation that the paper uses for results.","marker":"[11]"},{"why":"The GPT-4o-mini model that is instruction fine-tuned for Task-A and delivers the 0.9547 F1.","marker":"[9]"},{"why":"The BERT model fine-tuned for both tasks; it produces the best Task-B score of 0.4698.","marker":"[10]"},{"why":"The LoRA method used to fine-tune LLaMA-3 8B in 4-bit precision for the attribution task.","marker":"[14]"},{"why":"The low-code library used to fine-tune BERT with the reported hyperparameters.","marker":"[15]"},{"why":"Motivates Task-B as identifying the specific neural text generator, framing authorship attribution as a Turing test benchmark.","marker":"[2]"}],"fun_headline_variants":["One-line prompt is all GPT-4o-mini needs for 95% AI-text detection","GPT-4o-mini with one-line prompt hits 95% AI-text detection","95% detection with GPT-4o-mini, but AI authorship ID stays at 47%","AI-text detection: GPT-4o-mini reaches 95%, attribution only 47%","Fine-tuned GPT-4o-mini detects AI text at 95% accuracy"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The reported Task-A score assumes that the roughly 200 test samples blocked by the provider's content filter were handled in a way that does not inflate performance; if those samples were simply dropped, the 0.9547 F1 could be an upper bound rather than an unbiased estimate.","fun_headline_variants_meta":{"raw":{"variants":["One-line prompt is all GPT-4o-mini needs for 95% AI-text detection","GPT-4o-mini with one-line prompt hits 95% AI-text detection","95% detection with GPT-4o-mini, but AI authorship ID stays at 47%","AI-text detection: GPT-4o-mini reaches 95%, attribution only 47%","Fine-tuned GPT-4o-mini detects AI text at 95% accuracy"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000873,"raw_usage":{"total_tokens":3815,"prompt_tokens":1020,"completion_tokens":2795,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":636,"completion_tokens_details":{"reasoning_tokens":2677}},"tokens_in":636,"tokens_out":2795,"duration_ms":19814,"temperature":1.0,"reasoning_tokens":2677,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T19:30:51.272242+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the Task-A evaluation on the same test set with two rules for the ~200 content-filtered samples: count all of them as errors, and count all of them according to their true labels; if the macro F1 drops by more than a couple of points under the error rule, the claim of 0.9547 is not robust to the filtering artifact. The same recalculation should be repeated in a deployment setting with no content filtering applied at all.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the labeled train/validation/test texts for both tasks; the paper's reported F1 scores are measured on this dataset."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines the DeFactify task setup, labels, and macro F1 evaluation that the paper uses for results."},{"cited_title":"Accessed: 2024-07-18","cited_arxiv_id":null,"evidence_quote":"The GPT-4o-mini model that is instruction fine-tuned for Task-A and delivers the 0.9547 F1."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"The BERT model fine-tuned for both tasks; it produces the best Task-B score of 0.4698."},{"cited_title":"ktrain: A Low-Code Library for Augmented Machine Learning","cited_arxiv_id":"2004.10703","evidence_quote":"The low-code library used to fine-tune BERT with the reported hyperparameters."}],"review_version":1}