REVIEW 3 major objections 4 minor 22 references
QUST_NLP at SemEval-2025 Task 7: A Three-Stage Retrieval Framework for Monolingual and Crosslingual Fact-Checked Claim Retrieval
T0 review · 3 major / 4 minor · reviewed 2026-08-07 · deepseek-v4-flash
Pith's one-line read A three-stage pipeline—dense retrieval, cross-encoder re-ranking, and weighted voting—retrieves fact-checked claims with 93.64% Success@10 monolingually and 79.25% crosslingually at SemEval-2025 Task 7.
desk verdict Solid shared-task system with credible official results, but the claimed voting gains are in-sample on tiny dev sets. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The central mechanism is the three-stage pipeline. Stage one uses dense retrieval with a multilingual text encoder to cut a large claim pool down to 100 candidates per query. Stage two applies cross-encoder re-rankers (the BAAI bge-reranker family) to reorder those candidates. Stage three combines several re-rankers' top-10 lists through weighted voting, with each model's weight set to its S@10 on the development set. The paper also treats input configuration as part of the machinery: concatenating the original post, its machine translation, and optionally the fact-check verdict field changes retrieval effectiveness by language.
What would settle it
Run the published pipeline with equal voting weights instead of dev-fitted weights on a held-out test split; if equal-weight voting matches or beats the fitted weights, then dev-derived weighting is not what drives the reported gain. Alternatively, re-select the per-language retrieval models and voting weights on a different small dev sample; if the leaderboard scores drop below the single best re-ranker, the configuration choices are overfit.
Extended reading notes
Core claim
On the task's evaluation metric, Success@10 (S@10, whether any relevant claim appears in the top 10), the paper's configuration selects e5-mistral-7b-instruct as the first-stage retrieval encoder, feeds it original text plus translation (monolingual) or translation only (crosslingual), reranks the top-100 candidates with BAAI bge-reranker models, and then fuses the rerankers' outputs by weights fitted to dev-set scores. Fine-tuning rerankers on task data produces the largest single gains (up to +19.74% for v2-m3), and the weighted-voting stage adds about 1.4% (monolingual) and 3.8% (crosslingual) over the best individual reranker. The final official scores are 93.64% (5th of 28) for monolingual and 79.25% (7th of 29) for crosslingual retrieval.
Load-bearing premise
The entire reported effectiveness rests on the assumption that the development sets—especially the smallest ones, with 42 Thai posts and 78 Arabic posts—are representative enough to guide per-language model selection and voting-weight fitting without overfitting to those few examples.
Editorial extensions
If this is right
- Machine translation acts as a cheap and effective data augmentation for monolingual claim retrieval: adding translated text to the original post raises S@10 for most languages compared to either alone.
- For crosslingual retrieval, consistency of translation matters more than mixing sources: using only translated text beats mixing original and translated text, which suggests translation quality is the main lever.
- Fine-tuning a smaller re-ranker on task data can outperform a larger off-the-shelf re-ranker, so the framework's gains do not require the largest available models.
- Weighted voting over re-rankers improves the ensemble to at least the level of its best member across all tested languages, making it a dependable final stage.
- Because the pipeline is language-agnostic and relies only on translation for new languages, it can be extended to low-resource languages by translating training data (as the authors did for Polish and Turkish).
Reading between the lines
- Beyond the paper, the same retrieve-rerank-vote recipe could transfer to other multilingual retrieval tasks, such as question answering or duplicate detection, since none of its components are specific to fact-checked claims.
- The small development sets for Thai (42 posts) and Arabic (78 posts) mean the fitted voting weights are likely noisy; an untested alternative is reciprocal rank fusion, which requires no training and might generalize more stably.
- The paper attributes crosslingual weakness to translation quality; a testable extension is to compare the pipeline with an LLM-based translator or to filter translated candidates by confidence before retrieval.
- Since weighted voting helped in every language, the errors of the individual re-rankers appear complementary; measuring pairwise disagreement between re-rankers could predict when voting will help.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper describes QUST_NLP's participation in SemEval-2025 Task 7, a fact-checked claim retrieval shared task. The authors propose a three-stage framework: a retrieval stage that selects the best-performing pre-trained embedding model per language; a re-ranking stage that applies several BGE reranker models (some fine-tuned) to the top-100 candidates; and a weighted voting stage that combines the top-10 outputs of the rerankers, with weights derived from validation-set performance. The system reportedly achieved 5th place in the monolingual track (official S@10 = 93.64%) and 7th place in the crosslingual track (official S@10 = 79.25%). The paper also reports development-set ablations of retrieval models, input field combinations, reranker choices, and fine-tuning variants, and it releases the system code at a public repository.
Significance. The official shared-task ranking provides externally validated evidence that the overall system is competitive, and the release of system code is a concrete reproducibility contribution. The paper's three-stage architecture is a reasonable and practical engineering solution to a real multilingual retrieval problem, and the per-language analysis of input combinations and reranker fine-tuning is useful for practitioners. However, the paper's central internal claim—that the three-stage framework, and in particular weighted voting, improves over its best component—is currently supported only by development-set results obtained on the same data used to select models and fit voting weights. Given the very small development sets for several languages, this claim needs stronger out-of-sample evidence before the framework's contribution over its own best reranker can be accepted.
major comments (3)
- [§2.3, §4.3, Table 5] The claim that weighted voting improves over the best re-ranking model is an in-sample result. Section 2.3 states that each reranker's weight is assigned based on its validation-set performance, and Section 4.3 compares Voting S@10 against individual rerankers on the same development sets (Table 3: 95.14 vs 93.73 monolingual; Table 4: 84.05 vs 80.25 crosslingual). The official test results in Table 5 report only the final voting system per language, with no comparison to the best single reranker on the test set. Given the development-set sizes in Appendix Table A1 (42 Thai posts, 78 Arabic posts, 105 Malay posts), the reported gains of 1.41 and 3.8 points could easily be within selection noise. Please report the official test-set S@10 of the best single reranker versus voting per language (or at least averaged), so that the reader can verify whether the third stage actually contributed to the official scores.
- [§4.1, Tables 1 and 3] Per-language choices of retrieval models, input field combinations, and rerankers are made on development sets without variance estimates or statistical significance testing. In Table 1, many differences between configurations are a few percentage points, but a single post changes S@10 by up to 2.38 points for Thai (1/42) and 1.28 points for Arabic (1/78). The same issue affects the reranker comparisons in Table 3. Please add variance estimates, bootstrap confidence intervals, or repeated-run information, or explicitly acknowledge that the per-language selections may overfit the small development sets.
- [§4.2, Table 3] The text states that v2.5-gemma2-lightweight 'outperforms the other models, achieving an S@10 score of 92.74%' and attributes this to larger parameter size. This is internally inconsistent with Table 3, where v2-gemma-ft achieves 93.73% on average and v2-m3-ft achieves 92.57%, and with the same section's later statement that fine-tuned v2-gemma outperforms v2.5-gemma2-lightweight. Please correct the summary sentence so that it does not contradict the reported results.
minor comments (4)
- [§1] There is a typo in the Introduction: 'particulaedrly' should be 'particularly'.
- [§4.2] The phrase 'We utilize the models that performe well' contains a typo ('performe' should be 'perform').
- [Appendix A, Table A1] The column headers of Table A1 are unclear: 'Mono Post Fact_check train dev test train, dev test' mixes counts for posts and fact-check claims without distinct labels. Please separate 'Posts' and 'Fact-check claims' into clearly labeled sub-tables or columns.
- [§4.4] The phrase 'our approach achieved the highest score of 93.64% in the monolingual track' is ambiguous because 'highest score' could be read as the top score among all participants rather than the authors' own best configuration. Please rephrase, e.g., 'our best configuration achieved S@10 of 93.64%.'
Circularity Check
No significant circularity; the system is validated against an external shared-task benchmark and the internal tuning is standard validation practice.
full rationale
The paper's central effectiveness claim is its official SemEval-2025 Task 7 result, computed by the task organizers on the held-out test set (Table 5), which is external to the authors' own code and configuration choices. The three-stage pipeline is an engineering system description, not a derivation that reduces to its inputs. The only fitted quantities are per-language retrieval-model selection, reranker selection, and weighted-voting weights, all tuned on the development sets as explicitly stated in Section 2.3. Reporting that voting improves over the best reranker on those same development sets (Tables 3 and 4) is in-sample evaluation rather than a prediction, and the paper does not present those dev gains as official test outcomes. The self-citations (Jiang 2023; Wang et al. 2023b) are peripheral references to standard fine-tuning and ensembling practices and are not load-bearing evidence for the system's rank. The concern that the tiny development sets (42 Thai posts, 78 Arabic posts) may make model selection and weight fitting overfit is a legitimate robustness/correctness risk, but it is not an instance of circular reasoning: the official test set provides an independent, external check. No equation or construction in the paper makes the reported result equivalent to its fitted inputs, so no circularity is present.
Assumptions & free parameters
free parameters (4)
- Reranker voting weights =
Not reported
- Per-language input field configuration =
O, T, or O, T, V per language
- Number of candidates retrieved and final list size =
100 and 10
- Best retrieval model per language =
e.g., e5-mistral-7b-instruct for 6 languages, bge-mul-gemma2 for Malay, mul-e5-large-instruct for Arabic
assumptions (3)
- domain assumption The training and test distributions for each language are similar enough that supervised fine-tuning on the training set transfers to the test set.
- domain assumption Machine translation preserves enough meaning for effective crosslingual retrieval.
- domain assumption The S@10 metric is an appropriate measure of retrieval success.
Cite this review
Pith. "Pith review of QUST_NLP at SemEval-2025 Task 7: A Three-Stage Retrieval Framework for Monolingual and Crosslingual Fact-Checked Claim Retrieval." pith.science (2026). https://pith.science/paper/RW5QE5AL
@misc{pith2026250617272,
author = {Pith},
title = {Pith review of: QUST_NLP at SemEval-2025 Task 7: A Three-Stage Retrieval Framework for Monolingual and Crosslingual Fact-Checked Claim Retrieval},
year = {2026},
howpublished = {\url{https://pith.science/paper/RW5QE5AL}},
note = {Machine review of arXiv:2506.17272}
}
read the original abstract
This paper describes the participation of QUST_NLP in the SemEval-2025 Task 7. We propose a three-stage retrieval framework specifically designed for fact-checked claim retrieval. Initially, we evaluate the performance of several retrieval models and select the one that yields the best results for candidate retrieval. Next, we employ multiple re-ranking models to enhance the candidate results, with each model selecting the Top-10 outcomes. In the final stage, we utilize weighted voting to determine the final retrieval outcomes. Our approach achieved 5th place in the monolingual track and 7th place in the crosslingual track. We release our system code at: https://github.com/warmth27/SemEval2025_Task7
Figures
Reference graph
Works this paper leans on
-
[1]
Jianlv Chen, Shitao Xiao, Peitian Zhang, Kun Luo, Defu Lian, and Zheng Liu. 2024. https://arxiv.org/abs/2402.03216 Bge m3-embedding: Multi-lingual, multi-functionality, multi-granularity text embeddings through self-knowledge distillation . Preprint, arXiv:2402.03216
arXiv 2024
-
[2]
Menglong Cui, Pengzhi Gao, Wei Liu, Jian Luan, et al. 2025. Multilingual machine translation with open large language models at practical scale: An empirical study. arXiv preprint arXiv:2502.02481
arXiv 2025
-
[3]
Changjiang Gao, Hongda Hu, Peng Hu, Jiajun Chen, Jixing Li, and Shujian Huang. 2024. Multilingual pretraining and instruction tuning improve cross-lingual knowledge alignment, but only shallowly. arXiv preprint arXiv:2404.04659
arXiv 2024
-
[4]
Viktor Hangya, Hossain Shaikh Saadi, and Alexander Fraser. 2022. Improving low-resource languages in pre-trained multilingual language models. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 11993--12006
work page 2022
-
[5]
Kaiyu Huang, Fengran Mo, Xinyu Zhang, Hongliang Li, You Li, Yuanchi Zhang, Weijian Yi, Yulong Mao, Jinchen Liu, Yuzhuang Xu, et al. 2024 a . A survey on large language models with multilingualism: Recent advances and new frontiers. arXiv preprint arXiv:2405.10936
arXiv 2024
-
[6]
Yue Huang, Chenrui Fan, Yuan Li, Siyuan Wu, Tianyi Zhou, Xiangliang Zhang, and Lichao Sun. 2024 b . 1+ 1> 2: Can large language models serve as cross-lingual knowledge aggregators? arXiv preprint arXiv:2406.14721
work page Pith review arXiv 2024
-
[7]
Ye Jiang. 2023. Team qust at semeval-2023 task 3: A comprehensive study of monolingual and multilingual approaches for detecting online news genre, framing and persuasion techniques. arXiv preprint arXiv:2304.04190
work page Pith review arXiv 2023
-
[8]
Chankyu Lee, Rajarshi Roy, Mengyao Xu, Jonathan Raiman, Mohammad Shoeybi, Bryan Catanzaro, and Wei Ping. 2024. Nv-embed: Improved techniques for training llms as generalist embedding models. arXiv preprint arXiv:2405.17428
arXiv 2024
Show all 22 references
-
[9]
Zehan Li, Xin Zhang, Yanzhao Zhang, Dingkun Long, Pengjun Xie, and Meishan Zhang. 2023. Towards general text embeddings with multi-stage contrastive learning. arXiv preprint arXiv:2308.03281
2023 arXiv
-
[10]
Alexandre Magueresse, Vincent Carles, and Evan Heetderks. 2020. Low-resource languages: A review of past work and future challenges. arXiv preprint arXiv:2006.07264
2020 arXiv
-
[11]
Niklas Muennighoff, Thomas Wang, Lintang Sutawika, Adam Roberts, Stella Biderman, Teven Le Scao, M Saiful Bari, Sheng Shen, Zheng-Xin Yong, Hailey Schoelkopf, et al. 2022. Crosslingual generalization through multitask finetuning. arXiv preprint arXiv:2211.01786
2022 arXiv
-
[12]
Qiwei Peng, Robert Moro, Michal Gregor, Ivan Srba, Simon Ostermann, Marian Simko, Juraj Podroužek, Matúš Mesarčík, Jaroslav Kopčan, and Anders Søgaard. 2025. Semeval-2025 task 7: Multilingual and crosslingual fact-checked claim retrieval. In Proceedings of the 19th Internation...
2025
-
[13]
Mat \'u s Pikuliak, Ivan Srba, Robert Moro, Timo Hromadka, Timotej Smolen, Martin Melisek, Ivan Vykopal, Jakub Simko, Juraj Podrouzek, and Maria Bielikova. 2023. Multilingual previously fact-checked claim retrieval. arXiv preprint arXiv:2305.07991
2023 arXiv
-
[14]
Jirui Qi, Raquel Fern \'a ndez, and Arianna Bisazza. 2023. Cross-lingual consistency of factual knowledge in multilingual language models. arXiv preprint arXiv:2310.10378
2023 arXiv
-
[15]
Liang Wang, Nan Yang, Xiaolong Huang, Linjun Yang, Rangan Majumder, and Furu Wei. 2023 a . Improving text embeddings with large language models. arXiv preprint arXiv:2401.00368
2023 arXiv
-
[16]
Liang Wang, Nan Yang, Xiaolong Huang, Linjun Yang, Rangan Majumder, and Furu Wei. 2024. Multilingual e5 text embeddings: A technical report. arXiv preprint arXiv:2402.05672
2024 arXiv
-
[17]
Taihang Wang, Jianxiang Tian, Xiangrun Li, Xiaoman Xu, and Ye Jiang. 2023 b . Ensemble pre-trained multimodal models for image-text retrieval in the newsimages mediaeval
2023
-
[18]
Shitao Xiao, Zheng Liu, Peitian Zhang, and Niklas Muennighoff. 2023. https://arxiv.org/abs/2309.07597 C-pack: Packaged resources to advance general chinese embedding . Preprint, arXiv:2309.07597
2023 arXiv
-
[19]
Xiaoman Xu, Xiangrun Li, Taihang Wang, Jianxiang Tian, and Ye Jiang. 2024 a . Team qust at semeval-2024 task 8: A comprehensive study of monolingual and multilingual approaches for detecting ai-generated text. arXiv preprint arXiv:2402.11934
2024 arXiv
-
[20]
Yuemei Xu, Ling Hu, Jiayi Zhao, Zihan Qiu, Kexin XU, Yuqi Ye, and Hanwen Gu. 2024 b . A survey on multilingual large language models: Corpora, alignment, and bias. arXiv preprint arXiv:2404.00929
2024 arXiv
-
[21]
online" 'onlinestring :=
ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list...
-
[22]
write newline
" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...
Reviewed August 7, 2026 · model on record in the stance chip above.
Discussion (0). Sign in to comment.