{"id":"8c9e79c0-b67f-4a4b-99eb-81dbc55fca47","arxiv_id":"2501.09604","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"high","formal_verification":"none","parameter_count":4,"one_line_summary":"A new 60,000-article Bangla fake news dataset is introduced, but the headline result that LLMs outperform traditional classifiers is undermined by the paper's own external evaluation.","lead":"Researchers built a bigger Bangla-language fake news dataset with 60,000 news items and tested several AI models on it. The dataset could be useful, but the paper's claim that large language models beat traditional methods is contradicted by the authors' own external test results.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Paper's own external test set contradicts its headline claim: SVM (F1 91) beats BLOOM (F1 67) on the independent test set.","rationale":"The paper's central contribution is a dataset and a benchmark claim that LLMs (specifically BLOOM 560M) outperform traditional methods for Bangla fake news detection. The load-bearing evidence for this claim is the evaluation. The internal holdout results (Table 3) support the claim, but the paper's own independent test set (Table 4) directly reverses it: SVM achieves macro F1 91 while BLOOM achieves 67. This is not a subtle assumption about label quality; it is a direct contradiction within the paper's reported results. If the external test set is meaningfully independent and representative, the abstract's statement that BLOOM 'significantly outperforms traditional methods' is empirically false. The reader's weakest assumption about label correctness is reasonable, but it is secondary: even with perfect labels, the evaluation inconsistency stands. The dataset counts also fail to add up when summing Table 2, which compounds the credibility problem. These issues justify rejecting the paper, matching the reader's verdict, but the precise load-bearing concern is the evaluation contradiction rather than only the label-quality risk. A simple reproduction of Table 4 would settle it, making the concern concrete and testable.","tokens_in":11589,"tokens_out":3967,"duration_ms":36230,"concrete_test":"Download the released BanFakeNews-2.0 dataset and independent test set; retrain the SVM (All Features) and BLOOM 560M (QLoRA) models exactly as described in Sections 3 and 4, and evaluate on the external 1,000-article set. If the reproduced macro F1 scores are SVM approximately 91 and BLOOM approximately 67, the headline claim is contradicted. As a secondary check, sum the counts in Table 2 and compare to the stated 47,000 authentic, 13,000 fake, and 60,000 total items.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim that fine-tuned BLOOM 560M 'achieves the highest performance' and 'significantly outperforms traditional methods' (Abstract) is contradicted by the paper's own Table 4. On the independently curated 1,000-article test set, SVM with 'All Features' trained on BanFakeNews-2.0 reaches macro F1 91, while BLOOM reaches only 67 — a 24-point reversal of the internal-holdout rankings in Table 3 (SVM 86, BLOOM 89). Because Table 4 is explicitly presented as the external, 'rigorous' evaluation and cross-comparison, this internal contradiction is load-bearing: if the external test is valid, BLOOM does not outperform SVM, and the abstract's claim is false. The dataset arithmetic is also inconsistent: summing Table 2 category counts gives 45,839 authentic and 13,993 fake (total 59,832), not the stated 47,000/13,000/60,000. These numerical inconsistencies, independent of any assumption about label correctness, undermine the central claim.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper introduces BanFakeNews-2.0, an extension of the authors' earlier BanFakeNews dataset, claimed to contain 60,000 Bangla news articles (47,000 authentic and 13,000 fake) across 13 categories, together with a manually curated independent test set of 1,000 articles. The authors benchmark several models on this resource: TF-IDF character and word n-grams with linear SVM, five BERT-family transformers, and four LLMs fine-tuned with QLoRA, and report precision, recall, and F1 for authentic and fake classes as well as macro F1. The abstract's headline claim is that fine-tuned BLOOM 560M achieves the highest macro F1 of 89 and significantly outperforms traditional methods, and the paper further claims that models trained on BanFakeNews-2.0 generalize better than those trained on the original BanFakeNews. The manuscript also includes a brief limitations section acknowledging dataset imbalance and the future threat of generative-model fake news.","tokens_in":11856,"tokens_out":5297,"duration_ms":48641,"significance":"If the benchmark were reliable, the release of a 60,000-article Bangla fake news dataset with an independent test set would be a useful contribution for low-resource NLP, and the comparison of QLoRA-fine-tuned LLMs with traditional classifiers would interest the community. The authors' commitment to publicly release the dataset and models is commendable. However, the reported evaluations contain internal contradictions that prevent the results from being relied upon, and the headline claim about BLOOM's superiority is directly contradicted by the paper's own external evaluation.","major_comments":[{"comment":"The abstract and Introduction claim that fine-tuned BLOOM 560M 'achieves the highest performance, with a macro F1 score of 89' and 'significantly outperforms traditional methods.' Table 4 shows the opposite on the independently curated external test set: SVM (All) trained on BanFakeNews-2.0 reaches macro F1 91, while BLOOM trained on the same data reaches 67. Since Table 4 is presented as the rigorous cross-comparison for generalization, this internal contradiction is load-bearing and the headline claim is unsupported.","section":"Abstract and Section 5, Table 4"},{"comment":"The macro F1 values for the Majority and Random baselines are arithmetically impossible under the standard equal-weight macro averaging. The Majority baseline has per-class F1 of 88 for authentic and 0 for fake, so the macro average is 44, not the reported 78. The Random baseline has per-class F1s 61 and 30, giving 45.5, not 63. Unless a different averaging scheme is intended, the table's baseline numbers cannot be correct; if a different metric is used, it must be explicitly defined.","section":"Table 3"},{"comment":"The category counts in Table 2 sum to 45,839 authentic and 13,993 fake articles (59,832 total), not the stated 47,000 authentic, 13,000 fake, and 60,000 total. Either the table or the headline numbers are wrong. This discrepancy affects the central description of the resource and must be resolved before the dataset can be trusted. The claimed addition of 11,700 fake articles to the original 1,300 also yields 13,000, not 13,993.","section":"Table 2, Abstract, and Section 2"},{"comment":"The paper repeatedly relies on a 'manually curated independent test set of 1,000 news articles' for the external evaluation in Table 4, but Section 2 provides no details about how this test set was constructed, selected, or annotated. No inter-annotator agreement is reported for this test set, and the source list in Table 6 includes blogs and satirical sites without explaining how they were validated as providers of fake news. Without this information, the credibility of the external evaluation is unestablished.","section":"Section 2 and Abstract (external test set)"}],"minor_comments":[{"comment":"The sentence 'we filtered out duplicates and removed items with over 50% or 300 words of token overlap' is ambiguous; please specify whether the threshold is 50% overlap or 300 words, and how overlap is computed.","section":"Section 2"},{"comment":"The statement that 'alpha is used as a regularization parameter' is not accurate for QLoRA, where alpha is a scaling hyperparameter for the low-rank adaptation, not a regularization parameter in the usual sense.","section":"Section 3.3"},{"comment":"The dataset name appears as 'BanFakeNews-2' in Table 1 and as 'BanFakeNews-2.0' elsewhere; please use one consistent name throughout.","section":"Table 1"},{"comment":"The validation procedure should clarify whether hyperparameters were selected on the validation half before evaluating on the test half; as written, the split description does not rule out leakage of validation information into the reported test numbers.","section":"Section 4.1"},{"comment":"The column heading 'Mac. F1' should be spelled out as 'Macro F1' and the averaging scheme should be defined somewhere in the experimental setup.","section":"Table 4"}],"recommendation":"reject","confidential_remarks":"The paper's own external evaluation reverses its headline claim: SVM outperforms BLOOM on the independent test set. Combined with the arithmetic inconsistencies in Tables 2 and 3, the central claims cannot be accepted as they stand. The dataset may still be a useful resource, and the authors could consider resubmitting after correcting the arithmetic, defining the external test set construction, and reframing the conclusions to match the actual results."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The core resource here is genuinely useful: BanFakeNews-2.0 extends the existing Bangla fake news dataset with more fake articles, adds an independent 1,000-item test set, and the authors release data and code. The manual verification protocol is reasonable. If the data turns out to be clean, this is a real contribution for a low-resource language.\n\nBut the paper's central claim doesn't survive contact with its own tables. The abstract says fine-tuned BLOOM achieves the highest macro F1 (89) and significantly outperforms traditional methods. Table 4, the external evaluation the authors themselves call the rigorous cross-comparison, shows SVM at 91 and BLOOM at 67. That is a 24-point reversal. Either the external test is valid and the abstract is wrong, or the external test is not valid and the paper's main evaluation is broken. Either way, the headline claim as written is false.\n\nThe internal numbers are also inconsistent. In Table 3, the majority baseline shows per-class F1s of 88 and 0, which gives a macro F1 of 44, not the reported 78. The random baseline per-class F1s are 61 and 30, giving 45.5, not 63. Table 2 sums to 45,839 authentic and 13,993 fake, not the stated 47,000 and 13,000. These aren't minor typos; they indicate the reported results were not carefully checked.\n\nThe other concern is data provenance. The fake-news sources in Table 6 include blogs and satirical sites, not just established fact-checkers. The manual annotation with three annotators and 0.93 agreement is a plus, but the external test set's independence and label quality need scrutiny.\n\nSo what's the bottom line? The dataset idea is worth taking seriously. The benchmark methodology is standard, and the authors tried to do the right thing with an independent test set. But in its current form, the paper is not publishable: the arithmetic errors are pervasive, and the abstract contradicts the authors' own external results.\n\nI'd still send this to peer review rather than desk-reject. The resource has potential, and the flaws are fixable. A revised version with corrected numbers, an honest claim about BLOOM vs SVM, and a clearer discussion of the fake-news sources could be a solid contribution. As it stands, I wouldn't trust the tables, and I wouldn't cite it until the data and numbers are verified.","headline":"Useful dataset buried under self-contradictory numbers: the paper's own external test shows SVM beating BLOOM, and the tables don't add up.","tokens_in":12362,"tokens_out":1687,"would_cite":false,"duration_ms":18240,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A fine-tuned BLOOM 560M model reaches 89% macro F1 on Bangla fake-news detection, powered by a new 60,000-article dataset that the paper introduces.","keywords":["Bangla fake news","low-resource languages","fake news detection","BanFakeNews-2.0","BLOOM fine-tuning","QLoRA","transformer benchmark","dataset construction"],"falsifier":"Manually audit a random sample of the 13,000 fake articles by checking each against the fact-checking source cited in Table 6; if a substantial portion cannot be traced to a debunked claim, the dataset's labels do not support the reported F1 scores. Alternatively, compute n-gram or document-level overlap between the 1,000-article test set and the training portion; if the overlap is high, the external 'generalization' result in Table 4 is an artifact of test contamination.","tokens_in":11452,"feed_emoji":"📰","tokens_out":5943,"duration_ms":48114,"temperature":0.7,"pith_summary":"The paper introduces BanFakeNews-2.0, a 60,000-article Bangla fake-news dataset with 13,000 verified fake items and 47,000 authentic items across 13 categories. It argues that this larger, more balanced resource lets transformer models and fine-tuned large language models detect Bangla fake news substantially better than traditional linguistic-feature classifiers and better than models trained on the original BanFakeNews dataset. The best result is a fine-tuned BLOOM 560M model reaching 89% macro F1 on an independent 1,000-article test set, against 86% for the strongest SVM baseline. The paper also releases the dataset and models to support further work on low-resource fake-news detection.","feed_headline":"New Bangla dataset lifts fake-news detection to 89% F1","feed_subtitle":"Fine-tuned BLOOM 560M beats traditional models on the 60,000-article BanFakeNews-2.0 resource.","key_machinery":"The central object is BanFakeNews-2.0, a 60,000-article Bangla news corpus in which the fake class is built from articles debunked by fact-checking sites and the authentic class from 30 mainstream Bangladeshi news portals, with labels cross-checked by three annotators using majority voting (inter-annotator agreement 0.93). The other load-bearing mechanism is QLoRA fine-tuning, which adapts a 560M-parameter BLOOM model in 4-bit precision with low-rank adapters, letting a small language model reach an 89% macro F1 score. The dataset's category balance and duplicate removal (dropping items with over 50% token overlap) are what the paper credits for the improved generalization seen on the independent test set.","core_discovery":"BanFakeNews-2.0 extends the earlier BanFakeNews resource by adding 11,700 manually verified fake articles gathered from fact-checking platforms, producing a 60,000-item corpus with a 47,000-to-13,000 authentic-to-fake split. The paper claims that this proportional dataset, combined with an independently curated 1,000-article test set, enables fine-tuned transformer and LLM classifiers to outperform SVM models using TF-IDF, character n-grams, and FastText embeddings. Specifically, fine-tuning BLOOM 560M with QLoRA achieves an 89% macro F1 score, and models trained on BanFakeNews-2.0 generalize better on the external test set than models trained on the original BanFakeNews (macro F1 67 vs 29 for BLOOM, 91 vs 39 for SVM). The authors take this as evidence that dataset scale and balance, not just model architecture, are the key to moving low-resource languages from scarcity to capability.","pith_inferences":["Because the fake-news class is sourced from fact-checking and debunking sites, the dataset may under-represent fake stories that have not been publicly debunked; a live deployment would need to test on never-seen rumor types.","The 89% macro F1 hides a class asymmetry: authentic-news F1 is 96 while fake-news F1 is 81, so a more balanced fake/authentic split or a focus on the fake class could raise practical usefulness for moderation.","The QLoRA fine-tuning recipe (4-bit quantization, rank 8, alpha 32) is a parameter-efficient template that could be transferred to other low-resource languages with similar dataset construction.","Releasing the external test set is only useful if its independence is audited; a simple overlap check between test articles and training sources would make the claimed generalization much easier to verify."],"forward_implications":["A larger, more balanced Bangla fake-news corpus lets fine-tuned small LLMs reach 89% macro F1, compared with 86% for the best SVM baseline and 78% for a majority classifier.","Models trained on BanFakeNews-2.0 generalize better to a manually curated external test set than models trained on the original BanFakeNews (SVM: 91 vs 39 macro F1; BLOOM: 67 vs 29).","Character n-gram features (C3-C5) and word unigrams are stronger traditional baselines than bigrams or trigrams for Bangla fake-news detection.","The public release of dataset and models provides a reproducible benchmark for future Bangla fake-news detection research."],"supporting_citations":[{"why":"Supplies the original BanFakeNews dataset, its category taxonomy, and the train/test split convention that BanFakeNews-2.0 extends and benchmarks against.","marker":"Hossain et al., 2020"},{"why":"Provides the BLOOM 560M multilingual language model that the paper fine-tunes with QLoRA to achieve its best 89% macro F1 result.","marker":"Scao et al., 2022"},{"why":"Introduces the BERT architecture used for the m-BERT-c and m-BERT-unc transformer baselines.","marker":"Devlin et al., 2018a"},{"why":"Measures inter-annotator agreement, referenced to support the label reliability of the new dataset.","marker":"Fleiss, 1971"}],"fun_headline_variants":["Bangla fake-news detection hits 89% F1 with richer 60K dataset","BanFakeNews-2.0 balanced 60K corpus lifts Bangla detection to 89% F1","Dataset scale, not model alone, pushes Bangla fake-news F1 to 89%","From scarce to capable: 60K Bangla articles empower fake-news detection","LLMs beat SVMs on Bangla fake news after dataset upgrade to 60K"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the ground-truth 'fake' labels, drawn from fact-checking sites and blogs, are correct and that the 1,000-article external test set is genuinely independent and representative of the real Bangla news stream.","fun_headline_variants_meta":{"raw":{"variants":["Bangla fake-news detection hits 89% F1 with richer 60K dataset","BanFakeNews-2.0 balanced 60K corpus lifts Bangla detection to 89% F1","Dataset scale, not model alone, pushes Bangla fake-news F1 to 89%","From scarce to capable: 60K Bangla articles empower fake-news detection","LLMs beat SVMs on Bangla fake news after dataset upgrade to 60K"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000607,"raw_usage":{"total_tokens":2859,"prompt_tokens":1005,"completion_tokens":1854,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":621,"completion_tokens_details":{"reasoning_tokens":1735}},"tokens_in":621,"tokens_out":1854,"duration_ms":14262,"temperature":1.0,"reasoning_tokens":1735,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T19:50:49.556275+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Manually audit a random sample of the 13,000 fake articles by checking each against the fact-checking source cited in Table 6; if a substantial portion cannot be traced to a debunked claim, the dataset's labels do not support the reported F1 scores. Alternatively, compute n-gram or document-level overlap between the 1,000-article test set and the training portion; if the overlap is high, the external 'generalization' result in Table 4 is an artifact of test contamination.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the original BanFakeNews dataset, its category taxonomy, and the train/test split convention that BanFakeNews-2.0 extends and benchmarks against."}],"review_version":1}