{"id":"8a476c61-df37-41ca-b384-8aea3108342f","arxiv_id":"2504.20668","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"An LLM-based multilingual pipeline retrieves previously fact-checked claims, filters irrelevant ones, summarizes articles, and predicts veracity, evaluated across 23 languages.","lead":"The paper builds and tests a multilingual tool that searches a database of already fact-checked claims, uses an AI model to pick the relevant ones, summarizes them, and suggests whether a new social media post is true or false. Fact-checkers could use it to avoid re-checking claims that were already debunked, though the study is small and some results are not statistically significant.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The effort-reduction claim is unmeasured: filtration metrics (TNR 90.2% vs 86.3%) and a six-person satisfaction study do not show that fact-checkers work faster or better, especially since the tool UI still displays all non-relevant fact-checks.","rationale":"The reader's weakest assumption is the unmeasured jump from filtration metrics to effort reduction, and the full text supports this as the central vulnerability. The abstract's headline contribution is explicitly about reducing effort and streamlining fact-checking, yet no experiment measures task completion time, decision accuracy, or workload against a no-tool baseline. The human evaluation provides only satisfaction scores from six students and academics, a sample the paper itself acknowledges is not professional fact-checkers. Two additional details strengthen the concern: the tool's interface still displays all non-relevant fact-checks in a separate list for robustness, so the filtration does not remove them from the user's field of view, and the error analysis (Table 14) shows that for Mistral Large a quarter of posts lack any relevant fact-check in the filtered context, meaning the pipeline can actively mislead rather than merely withhold irrelevant information. I do not elevate the criteria-based retrieval's self-referential ground truth or the non-significant veracity prediction results to the primary concern because the abstract's central claim does not depend on those components; it depends on the effort-reduction inference. The automatic component evaluations are honestly reported, the dataset and code are released, and the filtration result is plausible as a precision gain, so a REJECT verdict would be too harsh. The correct stance remains CONDITIONAL: the effort-reduction claim needs a baseline-controlled, professionally sampled workload study before it can be accepted as stated.","tokens_in":31649,"tokens_out":3528,"duration_ms":35709,"concrete_test":"Run a controlled user study with at least 20 professional fact-checkers: for 30 claims with known relevant fact-checks, measure per-claim verification time, decision accuracy, and NASA-TLX workload in three conditions: (A) full ranked retrieval list, (B) LLM-filtered relevant-only list, and (C) the actual dual-list tool UI. Use paired significance tests on medians. If neither B nor C beats A on time or workload at equal-or-better accuracy, the abstract's effort-reduction claim should be removed or restricted to 'filtration improves precision.'","verdict_should_be":"UNCHANGED","load_bearing_attack":"The abstract makes a causal claim: LLM filtration 'reduce[s] effort and streamline[s] the fact-checking process.' The supporting evidence is Section 5's pair-level TNR/FNR (Mistral Large: TNR 90.2% vs 86.3% baseline, FNR 15.4%, S@10 dropping from 0.76 to 0.70) plus Section 8.2's satisfaction ratings. Neither measures fact-checker effort. A 15.4% FNR means roughly one in six relevant fact-checks is filtered out, and Table 14 shows 25.5% of posts have no ground-truth relevant fact-check in Mistral Large's filtered context, so the tool can send fact-checkers toward incomplete evidence. Furthermore, the tool UI (Appendix F.1) still shows all non-relevant fact-checks in a separate list; the LLM only relabels them. If fact-checkers must scan both lists to guard against misclassified relevant claims, the 3.9-point TNR gain may not translate into time saved. The human study has six non-professional participants, no baseline condition, and no accuracy or time metric; the paper's own Limitation section concedes this. The load-bearing assumption—that filtration quality improvements imply fact-checker effort reduction—is unmeasured.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a multilingual pipeline for retrieving previously fact-checked claims from social media posts. The pipeline has four stages: dense retrieval with text embedding models, LLM-based filtration that removes irrelevant fact-checks, LLM summarization of relevant fact-check articles, and LLM veracity prediction based on the selected fact-checks. The authors introduce the AFP-Sum dataset (~19K AFP fact-checking articles with human-written summaries across 23 languages), evaluate a battery of embedding models and LLMs on retrieval, filtration, summarization, and veracity prediction, and report a web-based tool with a small human evaluation. The headline results are that Mistral Large filtration raises the true-negative rate from 86.3% to 90.2% while keeping the false-negative rate at 15.4%, and veracity prediction improves from 26.5 to 63.1 Macro F1 when retrieved fact-checks are supplied.","tokens_in":31934,"tokens_out":4555,"duration_ms":47151,"significance":"If the central claims hold, this is a useful contribution to multilingual fact-checking support: the systematic comparison of TEMs across 20 languages, the new AFP-Sum dataset, the open code, and the transparent error analysis in Appendix E.1 are concrete assets. The paper also makes a practical contribution by shipping a working tool. However, the headline claim that LLM filtration 'reduces effort and streamlines the fact-checking process' is not supported by the measurements actually reported, and part of the evaluation (criteria-based retrieval in §4.2) is self-referential. These issues need to be fixed before the paper's main contribution is reliable as stated.","major_comments":[{"comment":"The abstract's causal claim that LLMs 'reduce effort and streamline the fact-checking process' is not supported by the experiments. Section 5.1 reports pairwise TNR/FNR on a candidate set and §8.2 reports satisfaction scores from six non-professional participants, with no measurement of task completion time, decision accuracy, or workload against a no-tool baseline. Moreover, the interface in §F.1 still lists all non-relevant fact-checks (Figure 10), so the LLM relabels information rather than removing it, and the reader cannot infer that effort is reduced. Either add a task-based evaluation that measures human performance or time, or remove the causal effort-reduction wording from the abstract, §9, and §10.","section":"Abstract, §5.1, §8.2, §F.1"},{"comment":"The criteria-based retrieval evaluation is circular. The ground truth is created by using Multilingual E5 Large to rank manually pre-filtered data, and the predicted list is produced by the same model using natural-language instructions. The negative Spearman/Kendall correlations therefore measure self-consistency between two prompt/input formulations of the same model, not the accuracy of criteria-based filtering. The conclusion that TEMs 'struggle' with date-range filtering needs independent relevance judgments or a reference ranking from a different model to be meaningful.","section":"§4.2, Table 3"},{"comment":"Table 14 shows that even for the best filtration model (Mistral Large), 25.5% of posts have no ground-truth relevant fact-check in the LLM-filtered context, and the authors acknowledge in §E.1 that retrieval quality is a bottleneck. This means the claimed reduction in fact-checker effort coexists with a substantial risk of presenting incomplete evidence. The unconditional claims in the abstract and §10 should be conditioned on this failure rate, and the discussion in §9 should explicitly state how a fact-checker using the tool should handle the missing-fact-check cases.","section":"§7.1, Table 14, §E.1"},{"comment":"The filtration experiments compare LLMs only against the Multilingual E5 Large baseline. Since §2 reviews prior LLM-based claim-matching and re-ranking approaches (e.g., Choi and Ferrara 2024a,b; Shliselberg and Dori-Hacohen 2022), at least one such method should be included as a comparative baseline. Without it, the claim that the proposed pipeline improves on existing claim-retrieval and filtration approaches is not established.","section":"§5.1"}],"minor_comments":[{"comment":"The sentence 'providing the article before the instruction significantly improved performance (p < 0.05), yielding better results when the article was provided after the instruction' is self-contradictory; the final phrase should presumably read 'before the instruction.'","section":"§6.1"},{"comment":"'We scrapped fact-checking articles' should be 'scraped.'","section":"§3.1"},{"comment":"'The analysis reseals variations' should be 'reveals variations.'","section":"§E.1"},{"comment":"The similarity threshold of 0.8 is introduced without justification; please state how this threshold was chosen and whether the results are sensitive to it.","section":"§4.2"},{"comment":"The paper mentions selecting an optimal threshold using Youden's Index for the binary classification framing but does not report the threshold value or how it was computed; please provide this detail for reproducibility.","section":"§5.1"},{"comment":"The column header 'Avg. Common FCs' is unclear; specify that it shows the mean proportion of fact-checks shared between the predicted and reference lists.","section":"Table 3"}],"recommendation":"major_revision","confidential_remarks":"The paper is a broad systems-oriented study with a useful dataset and a working tool, but its headline claim about effort reduction is not directly evidenced, and one evaluation block is self-referential. I would advise the editor to require either a modest human-in-the-loop experiment with time/accuracy measures or a substantial toning down of the abstract and conclusions before publication. The novelty is moderate—each pipeline component builds on prior work—so the contribution is mainly the integration, the multilingual evaluation, and the new summarization dataset, not a new learning method."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"This is a solid applied-NLP paper whose main value is a genuinely new dataset, AFP-Sum, and a broad multilingual evaluation of a fact-checking pipeline. The headline claim that LLM filtration 'reduces effort' is not directly measured; the evidence shows a better precision/recall trade-off, which is useful, but the effort-reduction bridge is unstated.\n\nThe AFP-Sum dataset—19K fact-checking articles with human-written summaries across 23 languages—is a real contribution, and the evaluation across 20 languages for retrieval and 23 for summarization is unusually broad. The filtration result is plausible: Mistral Large removes 90.2% of irrelevant fact-checks while missing 15.4% of relevant ones, versus 86.3% and 25.6% for the embedding baseline. That is a meaningful improvement in the trade-off, and the main filtration labels come from MultiClaim's human-derived links, so that evaluation is grounded.\n\nThe soft spot is exactly what the stress-test note flags. The abstract says this reduces effort and streamlines fact-checking, but no experiment measures time, accuracy, or workload. The six-person satisfaction study has no baseline and no task-based metrics. The tool's interface still shows all non-relevant fact-checks in a separate list, so a careful user may need to scan both lists, and the 15.4% FNR plus the ~25% of posts with no relevant fact-check in the filtered context are exactly the cases requiring that extra scan. This is an addressable gap: a baseline-controlled user study with a time or accuracy metric would either support or weaken the claim.\n\nThe criteria-based retrieval evaluation (Section 4.2) has a circularity: the ground-truth ranking is produced by the same Multilingual E5 Large model that is being evaluated, so those negative Spearman correlations are hard to interpret. That is a real but minor flaw. The summarization evaluation uses Google-translated references, which is standard practice, and the manual error analysis for veracity prediction is thoughtful and honest.\n\nOverall, the paper deserves a serious referee. The dataset and the scale of the multilingual comparison justify referee time, and the central filtration finding is believable. The authors are transparent about their limitations and release code and data. I largely agree with the reader's conditional verdict. This is for researchers working on multilingual fact-checking, claim retrieval, or applied LLM pipelines. I would send it to review and ask the authors to either tone down the effort-reduction claim or back it with direct evidence.","headline":"A solid multilingual fact-checking pipeline with a genuinely new dataset (AFP-Sum), but the abstract's effort-reduction claim is not directly measured and should be toned down or tested with a baseline-controlled study.","tokens_in":32479,"tokens_out":4457,"would_cite":true,"duration_ms":44392,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A multilingual pipeline retrieves past fact-checks, then an LLM filters the irrelevant ones, summarizes the rest, and predicts veracity with retrieved context.","keywords":["claim retrieval","fact-check filtering","veracity prediction","multilingual summarization","misinformation","text embedding models","large language models","social media claims"],"falsifier":"Measure whether the filtration step actually saves time: in a randomized trial with professional fact-checkers, compare time-to-decision and decision accuracy with the full candidate list versus the LLM-filtered list; if the 90% true-negative filtering does not reduce completion time or improve accuracy, the effort-reduction claim fails.","tokens_in":31458,"feed_emoji":"🔍","tokens_out":6259,"duration_ms":54646,"temperature":0.7,"pith_summary":"The paper argues that previously fact-checked claims, even in other languages, can be found and matched to a new social-media post without starting from scratch, and that a large language model can separate the truly relevant fact-checks from the merely similar ones. The proposed pipeline retrieves the top candidates with a multilingual embedding model, asks an LLM to keep only directly relevant fact-checks with explanations, summarizes each kept article in English, and then predicts the post's veracity from that evidence. Across ten languages and a thousand posts, the best LLM filter raised the true-negative rate from 86.3% to 90.2% while losing only 15.4% of genuinely relevant fact-checks, and veracity prediction with retrieved context more than doubled Macro F1 for the best model. The purpose is to let fact-checkers check whether a claim has already been debunked without reading every near-match.","feed_headline":"LLM filter lifts fact-check triage from 86% to 90%","feed_subtitle":"A four-step multilingual pipeline retrieves past fact-checks, summarizes them, and predicts veracity to lighten fact-checker workload.","key_machinery":"The carrying mechanism is a four-stage pipeline: (1) retrieval, where a multilingual text embedding model (Multilingual E5 Large) ranks the most similar previously fact-checked claims; (2) filtration, where a prompt instructs an LLM to keep only claims implied by the input, with an explanation for each; (3) summarization, where the LLM condenses each relevant fact-checking article into a 3-5 sentence English summary; (4) veracity prediction, where the LLM sees the input plus the kept claims, summaries, and ratings and outputs True/False/Unverifiable with justification. The filtration prompt is the load-bearing novelty: it converts a ranking problem into a relevance-classification problem that the LLM can justify, and its true-negative/false-negative trade-off determines everything downstream.","core_discovery":"The paper's central claim is that an LLM placed after a multilingual text-embedding retriever can serve as a reliable relevance gate for previously fact-checked claims, and that the gated set carries enough signal to summarize articles and predict veracity. Concretely, on a 1,000-post, ten-language subset of the MultiClaim collection, the strongest filter (Mistral Large) raised the true-negative rate from 86.3% to 90.2% while holding missed relevant fact-checks at 15.4%; the best veracity model with retrieved context reached a Macro F1 of 63.05 versus 26.53 without context. The paper also introduces a 23-language fact-check summarization corpus and shows that larger instruction-tuned models produce English summaries that survive ROUGE-L and BERTScore comparison against reference summaries. The intended consequence is that a fact-checker facing a new post can quickly see whether the claim was already debunked, in a language they may not read, and get a suggested verdict with explanation.","pith_inferences":["A testable extension the paper leaves open is a controlled time-and-accuracy study with professional fact-checkers; the current six-participant satisfaction study cannot confirm that the higher filtration rate shortens real workflow time.","The error analysis suggests a cheap improvement: because 27% of veracity errors came from missing context in the claim, feeding the LLM the full post text plus entity-level context before filtration could lift the downstream veracity score.","The same retrieval-plus-filtration pattern could transfer to other evidence-grounded tasks, such as medical misinformation triage or legislative claim tracking, wherever a database of previously vetted statements must be matched to a new utterance.","The 48.6% missing-fact-check rate for the smallest model is a warning that model choice, not retrieval architecture, may be the binding constraint for low-resource deployment."],"forward_implications":["With Mistral Large as the filter, the pipeline raises the true-negative rate from 86.3% to 90.2% while keeping the false-negative rate at 15.4%, so fact-checkers see markedly fewer irrelevant near-matches.","The LLM also produces English summaries and relevance explanations for the kept fact-checks across 23 languages, removing the language barrier for a checker working outside the original language.","Veracity prediction with retrieved fact-checks and summaries lifts Macro F1 from 26.53 to 63.05 for Mistral Large compared with the no-context baseline, making the suggestion usable as a starting point.","The whole pipeline runs with open-weight models (Llama3.3 70B) in the deployed tool, so the approach does not depend on a closed API for the core loop.","Because the LLM explains why each fact-check is relevant, the tool gives fact-checkers a reason to trust (or challenge) the filter rather than a black-box ranking."],"supporting_citations":[{"why":"Provides the MultiClaim dataset and the multilingual retrieval evaluation setup that this paper builds on and extends with LLM filtration.","marker":"Pikuliak et al., 2023"},{"why":"Establishes the fact-checker workload and verification needs that motivate the effort-reduction goal.","marker":"Hrckova et al., 2024"},{"why":"Prior work on LLMs for multilingual previously fact-checked claim detection that this pipeline extends to filtering, summarization, and veracity prediction.","marker":"Vykopal et al., 2025"},{"why":"Defines claim matching beyond English and underpins the cross-lingual retrieval objective.","marker":"Kazemi et al., 2021"},{"why":"Offers the textual-entailment alternative for LLM-based claim matching against which generative filtering is positioned.","marker":"Choi and Ferrara, 2024a,b"},{"why":"Supplies the generative re-ranking baseline for prioritizing previously fact-checked claims.","marker":"Shliselberg and Dori-Hacohen, 2022"}],"fun_headline_variants":["LLM filter raises fact-check true-negative to 90.2%","Retrieved context lifts veracity Macro F1 to 63.05","23-language fact-check summarization corpus from LLM pipeline","Multilingual LLM filter + summarizer: faster fact-check triage","LLM gate holds missed fact-checks at 15.4% while boosting triage"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that a higher true-negative rate in the filtered list translates into less human effort, but the paper never measures task time, workload, or decision accuracy against a no-tool baseline.","fun_headline_variants_meta":{"raw":{"variants":["LLM filter raises fact-check true-negative to 90.2%","Retrieved context lifts veracity Macro F1 to 63.05","23-language fact-check summarization corpus from LLM pipeline","Multilingual LLM filter + summarizer: faster fact-check triage","LLM gate holds missed fact-checks at 15.4% while boosting triage"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001369,"raw_usage":{"total_tokens":5543,"prompt_tokens":931,"completion_tokens":4612,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":547,"completion_tokens_details":{"reasoning_tokens":4515}},"tokens_in":547,"tokens_out":4612,"duration_ms":36811,"temperature":1.0,"reasoning_tokens":4515,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T05:23:02.253378+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Measure whether the filtration step actually saves time: in a randomized trial with professional fact-checkers, compare time-to-decision and decision accuracy with the full candidate list versus the LLM-filtered list; if the 90% true-negative filtering does not reduce completion time or improve accuracy, the effort-reduction claim fails.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Prior work on LLMs for multilingual previously fact-checked claim detection that this pipeline extends to filtering, summarization, and veracity prediction."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the generative re-ranking baseline for prioritizing previously fact-checked claims."}],"review_version":1}