{"id":"b041abde-b414-45c3-8c3f-22b870f62e1b","arxiv_id":"2412.15308","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"ViFactCheck provides the first publicly available multi-domain Vietnamese news fact-checking benchmark, with 7,232 labeled claim-evidence pairs, on which fine-tuned Gemma reaches 89.90% macro F1.","lead":"ViFactCheck is a new Vietnamese-language benchmark for news fact-checking: 7,232 human-annotated claim-evidence pairs labeled Support, Refute, or Not Enough Information, drawn from nine Vietnamese news outlets across 12 topics. The paper reports that fine-tuned Google Gemma reaches 89.90% macro F1 with gold evidence, the best of eleven models tested.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The dataset's own arithmetic is internally inconsistent: 1,000 articles times 6 claims per article cannot yield 7,232 claims, and the reported split ratios and context counts do not match. This must be resolved before the benchmark size and model rankings can be trusted.","rationale":"In good faith, the paper's intended contribution is a genuinely useful Vietnamese benchmark, and the model experiments are broadly plausible. However, the dataset's own provenance numbers do not cohere: 1,000 articles with six claims per article yields at most 6,000 claims, yet the paper claims 7,232, and the two reported split ratios contradict each other while the context counts match neither. This is more fundamental than the Kappa caveat: if the released dataset is not what the paper describes, the 89.90% headline and the claimed new standard are not interpretable. The reader's weakest assumption emphasizes label reliability with the 10% IAA sample; I agree that concern is real, but the arithmetic contradiction precedes it and can be checked immediately by downloading the released data. The proposed audit is a one-hour deterministic check that settles whether the concern lands. If the counts and splits check out, the remaining label-reliability and paraphrase-overlap concerns still warrant a larger audit but do not by themselves overturn the contribution. Therefore I keep the reader's CONDITIONAL verdict unchanged, with the condition that the dataset release and its reported statistics be verified and corrected as needed.","tokens_in":22735,"tokens_out":6519,"duration_ms":58033,"concrete_test":"Download the released repository at https://github.com/QuangDiy/ViFactCheck and run a deterministic audit: count distinct claim IDs, group claims by source article/context ID, and tally train/dev/test sizes. Then compare against the claimed 7,232 claims, the six-claims-per-article rule, and both split ratios. If the release does not contain 7,232 claims, the headline dataset count is wrong. If it does contain 7,232 claims, then Section 3.1, Appendix C, Section 4.2, and Appendix G must be corrected and the model results recomputed on the actual split before the 89.90% headline can be interpreted.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central dataset-size claim is internally inconsistent. Section 3.1 says the initial corpus contained 1,000 articles, and Appendix C says annotators generated six claims per article (two Support, two Refute, two NEI). These statements cap the dataset at 6,000 claims, yet the abstract, Section 1, and Table 9 report 7,232. Table 9 itself lists 2,289 contexts (1,035+496+758) and 7,232 claims (5,062+723+1,447), which would require roughly 3.16 claims per context and about 2,289 articles, not 1,000. The train/dev/test split is stated as 6:2:2 in Section 4.2 and as 7:1:2 in Appendix G; the claim totals match 7:1:2, while the context totals match neither. This is not a disagreement with external consensus but an internal contradiction in the paper's own description of the benchmark. Until it is resolved, the claimed dataset size, the partitioning, and every downstream model ranking built on that partition remain unverifiable.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"ViFactCheck is presented as the first publicly available multi-domain Vietnamese fact-checking benchmark. The authors describe collecting online news articles from nine Vietnamese outlets, generating claims labeled Support/Refute/NEI with annotated evidence, and validating labels via self-checking, cross-checking, and a Fleiss Kappa computation on 726 claims. They compare four pre-trained language models and several LLMs under fine-tuning and zero-shot prompting, using either full context or gold evidence, and report that fine-tuned Gemma achieves the best macro F1 of 89.90% with gold evidence. The paper also analyzes evidence retrieval, error types, topic and context-length effects, and training data size.","tokens_in":22757,"tokens_out":5357,"duration_ms":36545,"significance":"If the dataset statistics and split details are corrected and verified, ViFactCheck is a useful resource for Vietnamese NLP and low-resource fact-checking. It is one of the few Vietnamese datasets that covers multiple news domains and includes annotated evidence spans, and the public GitHub release of the dataset, checkpoints, and pipelines is a practical strength. The model comparison across PLMs, fine-tuned LLMs, and prompting methods is informative, and the evaluation is not circular: the reported F1 scores are measured on a held-out test set rather than quantities derived from the paper's own assumptions. However, the internal inconsistencies in the dataset size and partition ratios must be resolved before the benchmark's headline claims can be trusted.","major_comments":[{"comment":"The paper's own arithmetic on dataset size is internally inconsistent. Section 3.1 states that the initial corpus contained 1,000 articles, and Appendix C instructs annotators to generate six claims per article (two Support, two Refute, two NEI); that caps the dataset at 6,000 claims. Yet the abstract, Section 1, and Table 9 report 7,232 claims, and Table 9 lists 2,289 contexts (1,035+496+758). A 1,000-article corpus with six claims per article cannot produce 2,289 contexts or 7,232 claims. Please report the exact number of collected articles, the number of contexts after any filtering, the number of claims per article, and reconcile these totals.","section":"§3.1, Appendix C, Table 9"},{"comment":"The train/dev/test split is described inconsistently. Section 4.2 says the data were partitioned with a ratio of 6:2:2, while Appendix G states 7:1:2. The reported claim counts (5,062/723/1,447) match the 7:1:2 ratio (70/10/20%), but the reported context counts (1,035/496/758) match neither ratio. In addition, the paper never states whether the split is performed at the article/context level or at the claim level. This distinction matters for leakage: if claims derived from the same article appear in both training and test splits, the reported test performance may be optimistically biased. Please specify the split unit and give exact counts per subset.","section":"§4.2 vs. Appendix G"},{"comment":"The Fleiss Kappa of 0.83 is computed on only 726 claims, which is 10% of the 7,232 total, and these claims were relabeled by three annotators who did not author them. No inter-annotator agreement or independent validation is reported for the remaining 90% of the dataset. The abstract's characterization of the whole dataset as 'rigorously vetted' with a 0.83 reliability score is therefore stronger than the reported evidence. Please either report IAA on a larger sample or explicitly frame the 0.83 as an estimate from a 10% subsample, and preferably include per-label agreement.","section":"§3.3"},{"comment":"The headline macro F1 of 89.90% is obtained in the gold-evidence setting. Table 2 shows that Support claims share 63.52% Jaccard similarity and 86.89% related-word similarity with their associated evidence, and the Refute and NEI rows also show high overlap. This raises the concern that a substantial share of the gold-evidence performance may reflect lexical or paraphrase matching rather than verification reasoning. To support the claim that Gemma is effective at fact-checking, please report performance broken down by low-overlap versus high-overlap claim-evidence pairs, and temper the 'new standard' wording, which is also problematic because no existing Vietnamese fact-checking benchmark is compared directly.","section":"Table 2, Table 3, Section 4.3"}],"minor_comments":[{"comment":"The caption contains a typo: 'contruction process' should be 'construction process'.","section":"Figure 2 caption"},{"comment":"The text says the framework 'includes four SOTA open-source LLMs: Llama, Gemma, and Mistral', but only three names are listed; if the intended fourth is Gemini, the sentence should be corrected, and the label 'open-source' does not apply to Gemini.","section":"Section 4.1"},{"comment":"The description of Gemma states it was trained on 'up to 6 billion tokens', which appears inconsistent with the publicly available Gemma technical report; please verify the number and correct it if needed.","section":"Appendix H.2"},{"comment":"The column labeled 'F1 score' is ambiguous; please specify whether it is the macro F1 score and how it relates to the per-class Support/Refute/NEI columns.","section":"Table 5"},{"comment":"The statement that all performance differences are statistically significant at p < 0.01 is not accompanied by a description of the statistical test, the number of samples, or the multiple-comparison procedure; please provide these details.","section":"Table 3 footnote"}],"recommendation":"major_revision","confidential_remarks":"The central contribution is plausible and the artifact release is a strength, but the internal arithmetic inconsistencies (1,000 articles vs. 7,232 claims; 6:2:2 vs. 7:1:2; context counts matching neither ratio) are load-bearing for a benchmark paper. As editor, I would ask the authors to provide a single, consistent statistics table that sums to the abstract numbers and to confirm that the released GitHub artifacts match the corrected numbers before the manuscript is considered further."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"What you should know: ViFactCheck fills a real gap—a multi-domain Vietnamese news fact-checking benchmark with gold evidence spans is genuinely missing, and 7,232 claim-evidence pairs across 12 topics and 9 outlets, plus a solid baseline sweep, would be a useful resource. But the paper's own arithmetic doesn't hang together, and until that is fixed the headline numbers and model rankings can't be fully trusted.\n\nWhat is good: the construction design is thoughtful. Pilot annotation, detailed guidelines, self-checking and cross-checking, expert monitoring, and a Fleiss Kappa of 0.83 on a relabeled 10% subset all show care. The baseline study is competent: PLMs and LLMs, fine-tuning vs prompting, gold evidence vs full context, retrieval analysis (SBERT/BM25/hybrid, top-K), multi-evidence breakdown, and a useful error taxonomy. Artifacts are promised on GitHub with licenses listed. That is real work.\n\nSoft spots, in order of severity. First, the dataset size and split are internally inconsistent. Section 3.1 says 1,000 articles; Appendix C says six claims per article, which caps you at 6,000. The paper reports 7,232. Table 9 lists 2,289 contexts and 7,232 claims, which implies roughly 3.16 claims per context and about 2,289 articles. The split is stated as 6:2:2 in Section 4.2 and 7:1:2 in Appendix G; the claim counts match 7:1:2 exactly, while the context counts match neither. This is not a minor typo—it puts the benchmark's size and partitioning in question. Second, the 0.83 Kappa comes from 726 relabeled claims (10%) by three annotators; the remaining 90% has no independent verification. Self/cross-checking is good practice but doesn't substitute for a measured agreement. Third, the p<0.01 significance claim appears without any described test, and results are single runs, so the confidence intervals are unknown. Fourth, the 89.90% headline is in the gold-evidence setting, where Support claims share 63.5% Jaccard similarity with their evidence, so part of that score is paraphrase matching; calling it a \"new standard\" overreaches.\n\nNone of these are fatal to the resource's potential. They are fixable: report the actual article/claim counts, specify the split unit, describe the significance test, audit a larger label sample, and confirm the release. As written, I would not cite the dataset yet.\n\nFor whom: fact-checking and low-resource NLP researchers, especially those working on Vietnamese. It deserves a serious referee, but the review should be conditional on resolving the arithmetic and clarifying the annotation validation.\n\nRecommendation: send to peer review, major revision.","headline":"ViFactCheck fills a real gap for Vietnamese news fact-checking, but the paper's own statistics are internally inconsistent and need reconciliation before the benchmark size and model rankings can be trusted.","tokens_in":23556,"tokens_out":3870,"would_cite":false,"duration_ms":26872,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"ViFactCheck is the first publicly available benchmark for multi-domain Vietnamese news fact-checking, with 7,232 human-annotated claim-evidence pairs and a fine-tuned Gemma baseline reaching 89.90% macro F1 on gold evidence.","keywords":["fact-checking","Vietnamese NLP","benchmark dataset","claim verification","evidence-based","multi-domain news","large language models","low-resource languages"],"falsifier":"Replace each gold evidence paragraph in the test set with a similar-wording paragraph that does not support the claim, keeping the gold labels unchanged; if fine-tuned Gemma's macro F1 stays near 89.90%, the score reflects surface paraphrase overlap rather than genuine verification.","tokens_in":22300,"feed_emoji":"✔️","tokens_out":6147,"duration_ms":47006,"temperature":0.7,"pith_summary":"The paper introduces ViFactCheck, a new publicly available benchmark for fact-checking Vietnamese news claims across 12 topics and 9 online news outlets, built from 7,232 human-annotated claim-evidence pairs with high annotator agreement (Fleiss Kappa 0.83). It argues that this dataset fills a gap in low-resource fact-checking resources and demonstrates that fine-tuned language models, particularly Gemma, can reach strong performance, with a macro F1 of 89.90% when given gold evidence. The work matters because it provides a reusable testbed for developing and evaluating Vietnamese fact-checking systems, and it shows that the main difficulty lies in evidence retrieval rather than verification itself.","feed_headline":"New Vietnamese news fact-check dataset yields 89.9% F1 with Gemma","feed_subtitle":"7,232 human-annotated claims across 12 topics give low-resource fact-checking a reusable testbed.","key_machinery":"The central artifact is ViFactCheck itself: a dataset whose construction pipeline starts from 1,000 news articles from nine licensed Vietnamese outlets, from which annotators generate six claims per article (two per label) and mark the supporting evidence spans. The task is three-way classification of a claim as Support, Refute, or Not Enough Information (NEI) against a given context or evidence. The evaluation machinery is the comparison of fine-tuned and zero-shot prompted models under two input settings: full article context versus gold evidence spans; the paper's headline numbers come from macro F1 on a held-out test set. A further mechanism is the word-overlap and semantic-similarity analysis (LCS, NWR, Jaccard, SBERT cosine) used to argue that claim-context pairs are hard (low overlap) while claim-evidence pairs are easy (high overlap, 63.52% Jaccard for Support), explaining why gold evidence boosts scores.","core_discovery":"ViFactCheck is presented as the first publicly available benchmark dataset built specifically for multi-domain Vietnamese news fact-checking. The dataset was created by having seven native Vietnamese-speaking annotators generate claims from news articles and label them Support, Refute, or Not Enough Information, with explicit evidence annotations and a requirement that some claims combine multiple pieces of evidence; agreement on a 10% relabeled subset reached a Fleiss Kappa of 0.83. Evaluating fine-tuned and prompted models on this resource, the paper reports that fine-tuned Gemma achieves the highest macro F1 of 89.90% in the gold-evidence setting and 85.94% when given full article context, outperforming all tested multilingual and Vietnamese-specific baselines. The authors conclude that the dataset is reliable, that fine-tuning markedly outperforms prompting, and that gold evidence consistently beats full context, locating the hardest part of fact-checking in evidence retrieval.","pith_inferences":["The 63.52% Jaccard similarity between Support claims and their gold evidence suggests a large share of the 89.90% score may come from recognizing paraphrastic restatements rather than from multi-hop reasoning; spot-checking a random sample of correct predictions would reveal how much reasoning is actually happening.","The paper's reported train/dev/test split is inconsistent (6:2:2 in Section 4.2 versus 7:1:2 in Appendix G), and the table in Appendix G shows context counts that match neither ratio; before the dataset is used as a leaderboard, the canonical split needs to be fixed and documented.","The same annotation recipe—generate paraphrased claims, mark evidence spans, enforce multi-evidence examples—could be ported to other low-resource languages, giving a template for building fact-checking benchmarks where automatic translation of English datasets is not enough.","A natural extension would replace gold evidence with retrieved evidence at test time and measure end-to-end pipeline F1; based on the paper's own gap between full-context and gold-evidence scores, that end-to-end number would be considerably lower than 89.90%."],"forward_implications":["Vietnamese fact-checking now has a public, multi-domain testbed with 7,232 labeled claims, enabling consistent comparison of future systems.","Fine-tuned models with gold evidence far outperform both prompting and full-context inputs, meaning real gains in practice depend on building better evidence-retrieval pipelines.","Gemma-7B fine-tuned sets the current state of the art on this benchmark at 89.90% macro F1, a result the paper frames as establishing a new standard for the dataset.","Multi-evidence claims are measurably harder: all models drop in F1 when verification requires combining two or more evidence pieces.","Error analysis shows the residual failures concentrate in semantic ambiguity and complex inferential chains, giving a concrete target for future work."],"supporting_citations":[{"why":"Provides the foundational FEVER dataset and the Support/Refute/NEI task formulation that ViFactCheck adapts to Vietnamese news.","marker":"Thorne et al. 2018"},{"why":"MultiFC is the multi-domain evidence-based dataset this work models itself on and uses as comparison in Table 1.","marker":"Augenstein et al. 2019"},{"why":"ViWikiFC is the recent Vietnamese fact-checking benchmark this paper extends beyond Wikipedia to news.","marker":"Le et al. 2024"},{"why":"X-Fact motivates the low-resource and multilingual fact-checking gap the dataset aims to fill.","marker":"Gupta and Srikumar 2021"},{"why":"Supplies the Fleiss Kappa metric used to report the 0.83 inter-annotator agreement.","marker":"Fleiss 1971"},{"why":"Basis for the paper's claim that low word overlap between claim and context requires real inference ability.","marker":"McCoy, Pavlick, and Linzen 2019"},{"why":"Gemma, the model that achieves the headline 89.90% macro F1 after fine-tuning.","marker":"Team et al. 2024"},{"why":"SBERT embeddings drive the semantic-similarity analysis and the evidence-retrieval experiments.","marker":"Reimers and Gurevych 2019"}],"fun_headline_variants":["Vietnamese fact-check benchmark: Gemma hits 89.9% F1","First Vietnamese news fact-check dataset boasts 89.9% F1","Gemma tops new Vietnamese fact-check benchmark at 89.9 F1","New benchmark for Vietnamese news fact-checking: Gemma 89.9 F1","ViFactCheck: 7,232 Vietnamese claims, Gemma scores 89.9 F1"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The whole evaluation rests on the assumption that the 7,232 ground-truth labels are correct, but only 726 of them (10%) were ever checked by independent annotators, and the agreement statistic of 0.83 was measured on that small subset alone.","fun_headline_variants_meta":{"raw":{"variants":["Vietnamese fact-check benchmark: Gemma hits 89.9% F1","First Vietnamese news fact-check dataset boasts 89.9% F1","Gemma tops new Vietnamese fact-check benchmark at 89.9 F1","New benchmark for Vietnamese news fact-checking: Gemma 89.9 F1","ViFactCheck: 7,232 Vietnamese claims, Gemma scores 89.9 F1"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000485,"raw_usage":{"total_tokens":2409,"prompt_tokens":979,"completion_tokens":1430,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":595,"completion_tokens_details":{"reasoning_tokens":1323}},"tokens_in":595,"tokens_out":1430,"duration_ms":7066,"temperature":1.0,"reasoning_tokens":1323,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T11:52:40.731465+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Replace each gold evidence paragraph in the test set with a similar-wording paragraph that does not support the claim, keeping the gold labels unchanged; if fine-tuned Gemma's macro F1 stays near 89.90%, the score reflects surface paraphrase overlap rather than genuine verification.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the foundational FEVER dataset and the Support/Refute/NEI task formulation that ViFactCheck adapts to Vietnamese news."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"MultiFC is the multi-domain evidence-based dataset this work models itself on and uses as comparison in Table 1."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"X-Fact motivates the low-resource and multilingual fact-checking gap the dataset aims to fill."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Basis for the paper's claim that low word overlap between claim and context requires real inference ability."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"SBERT embeddings drive the semantic-similarity analysis and the evidence-retrieval experiments."}],"review_version":1}