{"id":"587b124d-5447-4510-aa88-2adda5fef823","arxiv_id":"2412.14276","paper_version":2,"verdict":"REJECT","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"high","formal_verification":"none","parameter_count":3,"one_line_summary":"On a GPT-4-plus-human-labeled fake news dataset, fine-tuned BERT and RoBERTa classifiers outperform instruction-tuned 7B LLMs, while LLMs are more robust to text perturbations.","lead":"This paper compares BERT-style models and large language models (LLMs) for fake news detection, using a 10,000-article dataset labeled by GPT-4 and reviewed by human experts. It reports that BERT-like models generally classify better, while LLMs are more robust to small text changes.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The evaluation labels are not validated as ground truth: with only ~72% inter-rater reliability and no independent gold standard, the reported F1 gap may measure agreement with GPT-4 bias rather than fake news detection.","rationale":"The paper's headline claim, that BERT-like models outperform autoregressive LLMs on a fake news detection task, is only as meaningful as the labels used for training and evaluation. The reader's weakest_assumption identifies exactly this: the GPT-4-plus-human labels are treated as ground truth without an independent gold standard. I agree that this is the most load-bearing concern. Section 3.1.3 reports an inter-rater reliability of approximately 72%, which is not chance-corrected and is modest for binary annotation; disagreements were settled by consensus, which can introduce anchoring or groupthink effects. No comparison between GPT-4 labels and final human labels is reported, so the degree of AI bias remaining in the data is unknown. Because the same labels are used both to fine-tune the classifiers and to compute the F1 scores in Table 6, any systematic labeling error directly contaminates the central comparison and the RQ2 label-quality comparison. The paper's internal contradiction in Section 5.1 (stating 'LLMs outperform BERT-like models' immediately after Table 6 shows the opposite) and the confounded RQ2 design are additional serious problems, but they are secondary to the question of whether the labels measure fake news at all. A concrete validation against an external fact-checked gold standard would settle the issue. If the labels fail validation, the paper's conclusions cannot be salvaged without re-annotating and re-running the experiments. If the labels pass validation, many of the other concerns, while still real, would be less decisive. Therefore the reader's REJECT verdict should remain unchanged.","tokens_in":19837,"tokens_out":7923,"duration_ms":75046,"concrete_test":"Release the dataset or a random sample (e.g., 500 articles) stripped of GPT/human labels. Have three independent professional fact-checkers, blind to the original labels, annotate each article with evidence-based veracity judgments, and compute chance-corrected agreement (Cohen's kappa or Fleiss' kappa) between the new gold standard and the paper's labels. If kappa is below 0.6, or if the paper's labels have materially different class balance, the ground-truth premise fails and the performance comparisons must be re-run against the new gold standard. If agreement is high, the label concern is resolved.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central comparison (Table 6) and the label-quality claim (RQ2) both depend on treating the GPT-4-plus-human labels as ground truth. Section 3.1.3 reports that eight reviewers reached only 'approximately 72%' agreement, resolved remaining disagreements by team consensus, and gives no chance-corrected kappa, no GPT-4-to-human agreement, and no validation against an independent fact-checked gold standard. For a binary decision, 72% raw agreement is modest; if label errors are systematic (e.g., political or stylistic bias of GPT-4), the F1 gap between RoBERTa and the LLMs, and the apparent superiority of AI labels over NELA-GT source labels, could reflect which model better fits the annotators' bias rather than which detects false information. Since the 10,000 labels are used for both training and test evaluation, a systematic labeling bias propagates into every reported metric. The paper's own data-availability statement says data are 'available upon request' rather than released, so the labels cannot currently be independently checked.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces a 10,000-article fake-news dataset labeled by GPT-4 Turbo with two-shot prompts and then reviewed by eight human reviewers, with disagreements resolved by team consensus. It fine-tunes three BERT-like models (BERT, DistilBERT, RoBERTa) and two 7B decoder-only LLMs (Llama-2-7B-Chat, Mistral-7B-Instruct) using QLoRA and instruction tuning, and compares them on this dataset and on a NELA-GT-2022 source-labeled subset. The main claims are that BERT-like models outperform LLMs on the GPT-labeled dataset (RoBERTa F1 89.68% vs. Mistral 80.23%), that LLMs are more robust to text perturbations, and that GPT-powered labels with human supervision yield better classifiers than source-level distant labels.","tokens_in":20025,"tokens_out":5624,"duration_ms":48702,"significance":"If the central claims held, the paper would provide a practically useful comparison: encoder-only models as efficient classifiers and decoder-only LLMs as more robust alternatives, together with a scalable human-supervised AI annotation pipeline. The paper has genuine strengths: it compares two model families under controlled fine-tuning, includes zero-/few-shot and instruction-tuned LLM variants, and makes code available. However, the significance is currently contingent on label validity and on the comparability of the two datasets, both of which are load-bearing and insufficiently established. The internal contradiction in the RQ1 answer and the unsupported perturbation claim further weaken the contribution in its present form.","major_comments":[{"comment":"The evaluation labels are not validated as ground truth. Section 3.1.3 reports only an inter-rater reliability score of 'approximately 72%' among eight reviewers, with no chance-corrected kappa, no GPT-4-to-human agreement, and no independent fact-checked gold standard. Since the same 10,000 labels are used for both training and test evaluation, any systematic labeling bias (e.g., GPT-4 stylistic or political bias) propagates into every reported metric in Table 6. The F1 gap between RoBERTa and the LLMs, and the superiority claimed for GPT labels over NELA-GT source labels, could reflect which model best fits the annotators' labeling bias rather than which model detects false information. The paper also states that data are 'available upon request' rather than released, so the labels cannot currently be independently checked. I would require an external validation component (e.g., evaluation on a fact-checked benchmark, reporting GPT-4/human agreement and kappa, or releasing the labels and annotation protocol) before the central comparison can be accepted.","section":"§3.1.3, Table 6"},{"comment":"The Discussion directly contradicts the paper's central claim. After presenting Table 6, in which RoBERTa (89.68 F1) and Mistral fine-tuned (80.23 F1) are the best models in each family, the text states: 'The results answer that for the task of fake news detection, LLMs outperform BERT-like models (RQ1).' This is the opposite of what Table 6, the abstract, and the rest of the paper report. This is not a minor stylistic slip because RQ1 is the main research question; the manuscript must state the answer consistently and unambiguously.","section":"§5.1, Discussion paragraph"},{"comment":"RQ2 is not identifiable from the current comparison. The GPT-labeled dataset and the NELA-GT-2022 subset differ in multiple dimensions simultaneously: collection period (May–Oct 2023 vs. calendar year 2022), news sources, minimum article length (150 vs. 50 words), train/validation/test split (80/10/10 vs. 80/20), and likely label distribution. Any performance difference in Table 6 could be due to any of these factors, not specifically to the labeling method. To support the claim that 'GPT-powered labels are more accurate than source-level labels,' the authors would need a matched comparison (e.g., the same articles labeled both ways, or article-level fact-check labels from NELA-GT), or at minimum an ablation that controls for source and time period.","section":"§5.1, Table 6, RQ2"},{"comment":"The robustness claim is based on a single article with 15 perturbation variants, and the table's own 'Correct per Model' row is inconsistent with the cells. Counting the rows, BERT and RoBERTa are predicted Fake on 6 of 15 perturbations (9 correct), DistilBERT on 5 (10 correct), and Llama-2-7B and Mistral-7B on 0 (15 correct), yet the table reports 10, 10, 10, 14, 14. Even if the counts were corrected, a single example cannot support a general conclusion about 'superior robustness,' and no repeated sampling, aggregate accuracy, variance, or significance testing is provided. The perturbation experiment needs to be run over a larger, representative sample of articles with statistical reporting before the RQ3 answer can be accepted.","section":"§5.4, Table 8"},{"comment":"The classification threshold is specified inconsistently. Section 4.3 states that 'a threshold of 0.8 is applied to these probabilities' for BERT-like models, while Table 3 lists 'Classification Threshold: 0.5 (default for all),' and §3.2.2 describes a confidence threshold of 0.8 for LLM majority voting. Since the decision threshold directly affects precision, recall, and F1, the paper must state which threshold was actually used for each model family and justify the choice. This is a load-bearing detail for all reported metrics.","section":"§4.3, Table 3, §3.2.2"}],"minor_comments":[{"comment":"The abstract says 'we developed an instruction-tuned LLM approach with majority voting during inference for label generation,' but in §3.2.2 majority voting is used for classification during inference, not for label generation. Please rephrase to avoid conflating annotation and classification.","section":"Abstract"},{"comment":"The text says 'The results in Table 4 demonstrate that increasing the number of demonstrations...' but Table 4 reports inference time, throughput, and training time; the relevant results are in Figure 4. Also, the text refers to '10 demonstrations per class,' which is not described in the experimental setup (only 5-shot is mentioned in §3.2.2).","section":"§5.3"},{"comment":"The numbers are inconsistent: §3.1.1 says about 30,000 articles were curated and 10,000 selected, while Table 2 says 'The curated dataset consists of 40,000 news articles labeled for fake news detection and the labeled dataset is 10,000.' Please reconcile.","section":"§3.1.1, Table 2"},{"comment":"Training time is contradictory: the text in §4.2 says each LLM was fine-tuned in 'approximately 50 minutes,' but Table 4 reports 3.0 hours for Llama-2-7B and 2.5 hours for Mistral-7B. Please correct one of these.","section":"§4.1, §4.2"},{"comment":"The ethics statement says 'no human participants were involved in this work,' but §3.1.3 describes eight human reviewers who annotated data. While this may not require REB approval under the stated criteria, the current phrasing is misleading and should be clarified or amended.","section":"Declarations"},{"comment":"The contribution states 'We release a dataset,' but the data availability statement says the data 'can be obtained from the first author.' For a dataset contribution to be verifiable, the data and annotation protocol should be released through a persistent repository or DOI.","section":"Declarations, Data availability"}],"recommendation":"major_revision","confidential_remarks":"The paper's central empirical claims are defensible in principle, but the current manuscript has several load-bearing problems: unvalidated ground-truth labels, a confounded RQ2 comparison, a direct contradiction in the RQ1 answer, and an unsupported perturbation claim with an internally inconsistent table. All of these are fixable with additional experiments and corrections, so I am not recommending rejection, but the revision will need to be substantive. In particular, I would like to see external label validation and a direct comparison that actually isolates label quality; without those, the main F1 comparison is not interpretable. The data-availability phrasing is also a concern for a dataset contribution in this venue."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: this is a real empirical study comparing BERT-like encoders and 7B LLMs for fake news detection, with a new 10k-article dataset labeled by GPT-4 plus human review. But the paper contains a direct internal contradiction in the Results section: the Discussion says LLMs outperform BERT-like models for RQ1, while Table 6 and the abstract say the opposite. That is not a minor typo; it is the paper's central finding. A reader cannot tell which result the authors actually stand behind.\n\nWhat the paper does well: the experimental setup is standard and mostly transparent. The authors fine-tune three encoders and two 7B LLMs with QLoRA, use 5-fold cross-validation, report precision/recall/F1, include a hyperparameter ablation for the BERT models, and compare against NELA-GT with source-level labels. Using GPT-4 with two demonstrations and then having eight human reviewers with consensus resolution is a reasonable annotation approach, and reporting the 72% inter-rater reliability is more honest than many papers would be.\n\nThe soft spots are load-bearing. First, the contradiction above. Second, RQ2 is confounded: the comparison between \"GPT labels\" and \"source-level labels\" changes the dataset, the articles, the time period, and the label definition at the same time, so any performance difference cannot be attributed to label quality. Third, the robustness claim rests on one article perturbed 15 ways in Table 8; that is illustrative, not evidence for a general conclusion about model families. Fourth, the labels are treated as ground truth without any independent gold standard; 72% raw agreement on a binary decision is moderate, and if GPT-4's labeling has systematic bias, the reported F1 gap may partly reflect which model agrees better with that bias. The paper's own data availability statement says the data are \"available upon request\" and the code link is a placeholder, so the central artifact cannot be independently checked.\n\nWho is this for? Practitioners choosing between cheap encoders and expensive LLMs for fake news detection, and people working on human-supervised LLM annotation pipelines. The paper could be valuable if revised, because the empirical scaffolding is there.\n\nMy recommendation: reject in its current form. The internal contradiction alone is disqualifying, and RQ2/robustness need redesign. But this is not a desk-reject-quality paper; it deserves a serious referee, with the expectation of major revision, not acceptance.","headline":"Competent applied comparison undermined by a direct results-section contradiction and unvalidated labels; the central claims as stated cannot be trusted.","tokens_in":20562,"tokens_out":2411,"would_cite":false,"duration_ms":22415,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that, on a GPT-4-annotated and human-reviewed dataset of 10,000 news articles, BERT-like encoder models beat autoregressive LLMs at fake-news classification, while LLMs are more robust to text perturbations.","keywords":["fake news detection","BERT-like models","large language models","GPT-4 annotation","human-in-the-loop labeling","distant supervision","text perturbation robustness","benchmark dataset"],"falsifier":"Take a random sample of a few hundred articles from this dataset and have independent professional fact-checkers label them, then compare those expert labels to the GPT-plus-human labels; if agreement is near chance, or if models retrained on the expert labels no longer show BERT-like models beating LLMs, the paper's central claim would be overturned. A second check is to run the same two model families on an established fact-checked benchmark such as LIAR and see whether the ordering holds.","tokens_in":19633,"feed_emoji":"📰","tokens_out":6801,"duration_ms":54554,"temperature":0.7,"pith_summary":"The paper is trying to establish two things: that encoder-only BERT-style models are the better classifiers for fake news, and that article-level labels produced by GPT-4 and then checked by human reviewers are a stronger training signal than the source-level distant labels commonly used in this field. On a new 10,000-article political-news dataset, fine-tuned RoBERTa reaches about 89.68% F1, while the best fine-tuned LLM, Mistral-7B, reaches about 80.23% F1. The result matters because it suggests that relatively small, efficient classifiers can still beat much larger generative models on a focused text-classification task, and that the quality of labels, not model size, may be the bottleneck. It also matters for annotation practice: generative AI with human oversight can produce usable training data without full expert annotation.","feed_headline":"RoBERTa beats fine-tuned LLMs on fake-news detection","feed_subtitle":"RoBERTa hits 89.7% F1 versus 80.2% for the best fine-tuned LLM on GPT-4-plus-human labels.","key_machinery":"The argument is carried by a specific annotation-and-evaluation pipeline. First, GPT-4 Turbo labels 10,000 political news articles as real or fake using a two-shot prompt; eight human reviewers independently re-check every label and resolve disagreements by consensus, yielding roughly 72% inter-rater agreement. Second, the comparison is run with two standardized model recipes: BERT-like models use a softmax classification head over the transformer output, while Llama-2-7B and Mistral-7B are instruction-fine-tuned with QLoRA and then prompted five times per article, with majority voting and an average-confidence threshold of 0.8 deciding the final label. The same two recipes are applied to the NELA-GT-2022 distant-supervision sample, which provides the weak-label control that lets the authors attribute part of the performance gap to label quality rather than only to model family.","core_discovery":"The central discovery is a performance ordering plus a data-quality finding. Fine-tuned BERT-like encoders, especially RoBERTa, outperform instruction-fine-tuned decoder-only LLMs on fake-news classification when both are trained on the same GPT-4-plus-human-labeled dataset; the paper reports RoBERTa at 89.68% F1 versus 80.23% for the best LLM. The authors also find that both model families score higher on this article-level human-supervised label set than on a matched sample of NELA-GT-2022 articles labeled by source-level distant supervision. In perturbation tests, the LLMs keep more of their accuracy when words are dropped, inserted, negated, or typo-ed, while the BERT-like models misclassify those modified examples more often.","pith_inferences":["The BERT-like advantage may partly reflect that GPT-4's labeling style is learnable by encoder models; if the labels encode stylistic tells, the ranking could shrink on independently expert-labeled datasets.","The absence of an independent gold standard means the roughly 72% inter-rater agreement leaves room for systematic label bias; a small expert re-labeling study would be the natural check.","A testable extension is to apply the human-supervised LLM-annotation pipeline to other classification domains, such as hate speech or stance detection, and see whether the same label-quality advantage appears.","The perturbation results suggest a cheap robustness recipe: keep a small encoder for clean inputs and route suspicious or edited items to an LLM, since the two model families fail in complementary ways."],"forward_implications":["If BERT-like models really are strong classifiers on this data, cost- and compute-constrained fake-news screening systems can rely on smaller encoders and reserve LLMs for harder cases.","If human-supervised GPT labels are superior to source-level distant labels, then article-level annotation with AI-plus-human review is a viable path to better training data than weak supervision.","The robustness gap suggests that a deployment combining a precise BERT-like classifier with an LLM-based check for perturbed or adversarial inputs could cover both accuracy and resilience.","Fine-tuning beats zero-shot and few-shot prompting for both Llama-2 and Mistral on this task, so task-specific adapters matter more than scale when the labels are good."],"supporting_citations":[{"why":"the GPT-4 Turbo model whose two-shot labels form the initial annotation of the 10,000-article dataset","marker":"[73]"},{"why":"the NELA-GT-2022 corpus whose source-level distant labels serve as the weak-supervision comparison condition","marker":"[78]"},{"why":"defines Mistral-7B, one of the two LLM families instruction-fine-tuned and evaluated in the study","marker":"[20]"},{"why":"defines Llama-2-7B, the other LLM family fine-tuned and evaluated in the study","marker":"[21]"},{"why":"supplies the QLoRA 4-bit quantization method used to fine-tune the LLMs under GPU memory limits","marker":"[76]"},{"why":"the evidence cited for the premise that LLM annotations can be more reliable than crowd-sourced ones","marker":"[17]"},{"why":"defines distant supervision, the labeling paradigm represented by the NELA-GT comparison","marker":"[16]"}],"fun_headline_variants":["RoBERTa beats fine-tuned LLMs on fake-news detection","Human-supervised AI labels boost fake-news classification","BERT-like models outsmart LLMs on fake-news tests","RoBERTa tops LLMs in fake-news F1: 89.7 vs 80.2","Fake-news AI: BERT beats GPT, humans improve labels"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the GPT-4-generated labels, after consensus review by eight reviewers, are correct enough to serve as ground truth for both training and testing; with roughly 72% inter-rater agreement and no independent gold standard, a systematic labeling error would change the rankings.","fun_headline_variants_meta":{"raw":{"variants":["RoBERTa beats fine-tuned LLMs on fake-news detection","Human-supervised AI labels boost fake-news classification","BERT-like models outsmart LLMs on fake-news tests","RoBERTa tops LLMs in fake-news F1: 89.7 vs 80.2","Fake-news AI: BERT beats GPT, humans improve labels"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000437,"raw_usage":{"total_tokens":2195,"prompt_tokens":893,"completion_tokens":1302,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":509,"completion_tokens_details":{"reasoning_tokens":1204}},"tokens_in":509,"tokens_out":1302,"duration_ms":9492,"temperature":1.0,"reasoning_tokens":1204,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T12:22:10.892816+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a random sample of a few hundred articles from this dataset and have independent professional fact-checkers label them, then compare those expert labels to the GPT-plus-human labels; if agreement is near chance, or if models retrained on the expert labels no longer show BERT-like models beating LLMs, the paper's central claim would be overturned. A second check is to run the same two model families on an established fact-checked benchmark such as LIAR and see whether the ordering holds.","supporting_citations":[{"cited_title":"In: Proceedings of the Joint Conference of the 47th Annual Meeting of the ACL and the 4th International Joint Conference on Natural Language Processing of the AFNLP, pp","cited_arxiv_id":null,"evidence_quote":"defines distant supervision, the labeling paradigm represented by the NELA-GT comparison"},{"cited_title":"https://openai.com/index/gpt-4/","cited_arxiv_id":null,"evidence_quote":"the GPT-4 Turbo model whose two-shot labels form the initial annotation of the 10,000-article dataset"},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"the NELA-GT-2022 corpus whose source-level distant labels serve as the weak-supervision comparison condition"}],"review_version":1}