REVIEW 2 major objections 6 minor 38 references
EVENT-Retriever: Event-Aware Multimodal Image Retrieval for Realistic Captions
T0 review · 2 major / 6 minor · reviewed 2026-08-05 · deepseek-v4-flash
Pith's one-line read The paper claims a four-stage, zero-shot pipeline—dense article retrieval, LLM reranking, VLM image scoring, and rank fusion—achieves the top-1 score in the EVENTA 2025 Track 2 image retrieval challenge.
desk verdict A clean challenge report with a credible top-1 result, but the load-bearing pool-coverage assumption is never directly measured. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing mechanism is the article gate. Stage 1 embeds each article's title, date, and content with Qwen3-Embedding and retrieves the top 10 articles by cosine similarity; Stage 2 gives each article a scalar relevance score from Qwen3-Reranker by computing the log-probability of the token 'yes' in a prompted yes/no judgment. The pipeline then draws candidate images only from these articles (at least 10 images from at least 3 distinct articles among the top 10), so every later image-level score inherits whatever the article stage got right or wrong. Rank-aware selection keeps at most three top images per article, and Reciprocal Rank Fusion combines the ranked lists from several config
What would settle it
Recompute the official private-test metric on a fresh sample from the same dataset; if the ensemble's score falls materially below the reported 0.5766, the top-1 claim was specific to the original leaderboard queries rather than a general property of the pipeline.
Extended reading notes
Core claim
On the paper's own terms, the core discovery is that event-based image retrieval in the open-domain news setting is a retrieval problem that should be decomposed by modality and by scale: retrieve articles with a multilingual text embedder (Qwen3-Embedding), rerank those articles with a prompted LLM (Qwen3-Reranker) that answers 'yes/no' to whether an article supports the caption, collect candidate images only from the top-ranked articles, score image-caption pairs with a multimodal embedder (gme-Qwen2-VL), and merge several such runs with Reciprocal Rank Fusion. The paper shows this ensemble reaching an overall score of 0.5766 on the private test set, ahead of every other participant and of
Load-bearing premise
The pipeline assumes the article retriever puts the article that contains the correct image inside the top 10 for almost every caption, because images are only collected from those top articles; on a 1000-caption training sample its Recall@10 is only 0.77.
Editorial extensions
If this is right
- If the article gate works, the system can retrieve images whose caption mentions no visible object, because the event context comes from the article text.
- The reported gap between the RRF ensemble (0.5766) and the best single run (0.5712) implies that combining configurations by rank is an inexpensive performance source, separate from model quality.
- The method's zero-shot design implies competitive event retrieval does not require supervised fine-tuning on the target corpus, which matters when the corpus is large or changes.
- Because image candidates come only from top articles, article retrieval quality sets the ceiling for the whole task; improving Recall@10 would directly bound end-to-end improvement.
Reading between the lines
- If article-level recall is the bottleneck, the clearest next step is to widen the candidate pool beyond the current top-10 (or add a hybrid lexical/dense search), since a quarter of training-like captions never reach the image stage.
- The paper's rank-aware cap of three images per article implicitly assumes the correct image rarely appears alongside many near-duplicates in the same article; on articles with 30+ images, that assumption could hide correct images below the cutoff.
- The success of RRF across different model scales suggests that ensemble diversity—not just single-model accuracy—may be doing much of the work; a testable extension is ablating configurations of equal quality but different scales.
- The paper's own limitation note points toward end-to-end training of caption-to-(article, image) as a natural next step; a joint model could compensate for the article stage's errors instead of inheriting them.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The manuscript describes EVENT-Retriever, a zero-shot four-stage retrieval pipeline for the EVENTA 2025 Track 2 challenge: Qwen3-Embedding dense article retrieval, Qwen3-Reranker prompt-based reranking, Qwen2-VL/GME image scoring with rank-aware selection, and Reciprocal Rank Fusion ensembling. The paper reports Recall@10=0.77 for caption-to-article retrieval, strong in-article caption-to-image results for gme-Qwen2-VL-2B-Instruct, and a final private-leaderboard overall score of 0.5766, which the authors state is top-1. The system is purely zero-shot and all components are open-source; code is released.
Significance. If the reported result holds, the paper demonstrates an effective modular recipe for long-form, event-centric image retrieval using Qwen-family models. Its strengths are the clean internal evaluations: Table 1 compares embedding/reranker combinations on 1000 training captions, and Table 2 compares vision-language models on 20K+ in-article queries. The release of code and the detailed prompt description support reproducibility. The principal weakness is that the headline claim rests on a single private-leaderboard score with a tiny margin (0.0044), and the two pipeline components responsible for the final gain—the image-collection early stopping and the RRF ensemble composition—are not analyzed or ablated.
major comments (2)
- [Section 3.3 / Section 4.2] Candidate-pool coverage is not evaluated. The traversal stops once I=10 valid images from A=3 distinct articles among the top-K are collected. This can exclude the ground-truth image even when the ground-truth article is in the top-K, simply because earlier articles fill the quota. Table 1's Recall@10=0.77 is an upper bound on article retrieval, not a pool-coverage rate; Table 4's final R@10=0.744 is a lower bound that conflates pool misses with ranking errors. The paper states that A, I, K are empirically tuned (Sec. 4.2) but reports no ablations over A/I, no direct pool-recall measurement, and no analysis of how often early stopping drops the correct article. A direct measurement of candidate-pool recall, plus ablations over A and I, is needed to show the thresholds are robust rather than accidental.
- [Section 4.5 / Table 4] The top-1 claim rests on a margin of 0.0044 over the second-ranked external system (0.5766 vs. 0.5722) on a single private leaderboard, with no error bars, significance tests, or variance estimates. In addition, the RRF ensemble is not specified: Section 3.4 states that it combines 'multiple high-performing runs,' but the paper never lists the individual runs, their hyperparameters, or their individual scores. Without an ensemble ablation (e.g., performance of each constituent run and a leave-one-out analysis), the reader cannot evaluate whether the RRF gain is systematic or a selection artifact.
minor comments (6)
- [Section 3.3] Heading contains a typo: 'Rank-A ware Image Selection' should be 'Rank-Aware Image Selection.'
- [Throughout] Naming is inconsistent: 'OpenEvent V1' (Sec. 1) vs. 'OpenEvents V1' (Sec. 4.1); 'Qwen3-Embed' vs. 'Qwen3-Embedding' are used interchangeably.
- [Table 3] The row label 'BGE-Reranker-v2-m3+ +' contains a spurious extra plus sign; also, the row labels are very long and could be abbreviated.
- [Table 2] R@10 for gme-Qwen2-VL-2B-Instruct is reported as 1.0000. Given finite data, this exact value is likely rounded; please report the actual value or a confidence interval.
- [Section 4.2] The selection of the '1000 representative event captions' from the training set is not described. The sampling procedure should be specified to assess representativeness.
- [Section 3.2] The multi-line prompt example is presented as prose; a formal listing or figure would improve readability and reproducibility.
Circularity Check
No significant circularity: the top-1 claim is an external benchmark result, not an input-equivalent derivation.
full rationale
The paper is a challenge system description, and its central claim is an externally evaluated ranking on the EVENTA 2025 private test set, not a derivation from first principles. Each pipeline stage (dense article retrieval with Qwen3-Embedding, Qwen3-Reranker prompted scoring, Qwen2-VL+GME image scoring, and RRF ensembling) is tested against held-out challenge data. The only self-references are to OpenEvents V1 [22] and the EVENTA challenge [26], which the same research group helped create; however, the private-leaderboard scores are produced by the challenge evaluation protocol and are not algebraically implied by the paper's equations or by fitting any parameter. The A=3, I=10, K=10 hyperparameters are claimed to be empirically tuned but no ablation is shown (Sec. 4.2), which is a reproducibility/validation gap rather than circularity: the final score is not defined in terms of those hyperparameters. The RRF ensemble aggregates ranked lists from different internal configurations; it does not fit a parameter to the target metric and then rename that fit as a prediction. No self-citation is load-bearing for the central top-1 result, and no uniqueness theorem or ansatz is imported from the authors' prior work. Thus no circular step is present.
Assumptions & free parameters
free parameters (4)
- Image collection hyperparameters (I, A, K) =
I=10, A=3, K=10
- Max images per article in rank-aware selection =
3
- RRF smoothing constant k =
60
- RRF run set =
not disclosed
assumptions (4)
- domain assumption Pretrained Qwen3-Embedding, Qwen3-Reranker, and gme-Qwen2-VL-2B-Instruct are zero-shot effective for the article and image relevance needed here.
- domain assumption The OpenEvents V1 dataset and EVENTA 2025 challenge metrics are a valid measure of event-based image retrieval.
- domain assumption The 'yes/no' logit from Qwen3-Reranker under the provided prompt is a reliable scalar relevance score.
- standard math Reciprocal Rank Fusion improves retrieval over single runs.
Cite this review
Pith. "Pith review of EVENT-Retriever: Event-Aware Multimodal Image Retrieval for Realistic Captions." pith.science (2026). https://pith.science/paper/2YATNMQJ
@misc{pith2026250900751,
author = {Pith},
title = {Pith review of: EVENT-Retriever: Event-Aware Multimodal Image Retrieval for Realistic Captions},
year = {2026},
howpublished = {\url{https://pith.science/paper/2YATNMQJ}},
note = {Machine review of arXiv:2509.00751}
}
read the original abstract
Event-based image retrieval from free-form captions presents a significant challenge: models must understand not only visual features but also latent event semantics, context, and real-world knowledge. Conventional vision-language retrieval approaches often fall short when captions describe abstract events, implicit causality, temporal context, or contain long, complex narratives. To tackle these issues, we introduce a multi-stage retrieval framework combining dense article retrieval, event-aware language model reranking, and efficient image collection, followed by caption-guided semantic matching and rank-aware selection. We leverage Qwen3 for article search, Qwen3-Reranker for contextual alignment, and Qwen2-VL for precise image scoring. To further enhance performance and robustness, we fuse outputs from multiple configurations using Reciprocal Rank Fusion (RRF). Our system achieves the top-1 score on the private test set of Track 2 in the EVENTA 2025 Grand Challenge, demonstrating the effectiveness of combining language-based reasoning and multimodal retrieval for complex, real-world image understanding. The code is available at https://github.com/vdkhoi20/EVENT-Retriever.
Figures
Reference graph
Works this paper leans on
-
[1]
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Floren- cia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. 2023. Gpt-4 technical report. arXiv preprint arXiv:2303.08774 (2023)
arXiv 2023
-
[2]
Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, et al. 2023. Qwen technical report. arXiv preprint arXiv:2309.16609 (2023). EVENT-Retriever: Event-Aware Multimodal Image Retrieval for Realistic Captions MM ’25, October 27–31, 2025, Dublin, Ireland
arXiv 2023
-
[3]
Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. 2023. Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyond.arXiv preprint arXiv:2308.12966 (2023)
arXiv 2023
-
[4]
Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, Humen Zhong, Yuanzhi Zhu, Mingkun Yang, Zhaohai Li, Jianqiang Wan, Pengfei Wang, Wei Ding, Zheren Fu, Yiheng Xu, Jiabo Ye, Xi Zhang, Tianbao Xie, Zesen Cheng, Hang Zhang, Zhibo Yang, Haiyang Xu, and Junyang Lin. 2025. Qwen2.5-VL Technical Rep...
arXiv 2025
-
[5]
Jianlv Chen, Shitao Xiao, Peitian Zhang, Kun Luo, Defu Lian, and Zheng Liu. 2024. Bge m3-embedding: Multi-lingual, multi-functionality, multi-granularity text embeddings through self-knowledge distillation. arXiv preprint arXiv:2402.03216 (2024)
arXiv 2024
-
[6]
Jianlyu Chen, Shitao Xiao, Peitian Zhang, Kun Luo, Defu Lian, and Zheng Liu
-
[7]
Yen-Chun Chen, Linjie Li, Licheng Yu, Ahmed El Kholy, Faisal Ahmed, Zhe Gan, Yu Cheng, and Jingjing Liu. 2020. Uniter: Universal image-text representation learning. In European conference on computer vision . Springer, 104–120
work page 2020
-
[8]
Zhe Chen, Weiyun Wang, Hao Tian, Shenglong Ye, Zhangwei Gao, Erfei Cui, Wenwen Tong, Kongzhi Hu, Jiapeng Luo, Zheng Ma, et al. 2024. How far are we to gpt-4v? closing the gap to commercial multimodal models with open-source suites. Science China Information Sciences 67, 12 (2024), 220101
work page 2024
Show all 38 references
-
[9]
Gordon V Cormack, Charles LA Clarke, and Stefan Buettcher. 2009. Reciprocal rank fusion outperforms condorcet and individual rank learning methods. In Proceedings of the 32nd international ACM SIGIR conference on Research and development in information retrieval. 758–759
2009
-
[10]
Arnon Dagan, Ido Guy, and Slava Novgorodov. 2023. Shop by image: Charac- terizing visual search in e-commerce. Information Retrieval Journal 26, 1 (2023), 2
2023
-
[11]
Venkat N Gudivada and Vijay V Raghavan. 1995. Content based image retrieval systems. Computer 28, 9 (1995), 18–22
1995
-
[12]
Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoc Le, Yun-Hsuan Sung, Zhen Li, and Tom Duerig. 2021. Scaling up visual and vision- language representation learning with noisy text supervision. In International conference on machine learning . PMLR, 4904–4916
2021
-
[13]
Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. 2023. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. In International conference on machine learning . PMLR, 19730–19742
2023
-
[14]
Manling Li, Ruochen Xu, Shuohang Wang, Luowei Zhou, Xudong Lin, Chenguang Zhu, Michael Zeng, Heng Ji, and Shih-Fu Chang. 2022. Clip-event: Connecting text and images with event structures. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition . 1...
2022
-
[15]
Xiaoqing Li, Jiansheng Yang, and Jinwen Ma. 2021. Recent developments of content-based image retrieval (CBIR). Neurocomputing 452 (2021), 675–689
2021
-
[16]
Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. 2014. Microsoft coco: Common objects in context. In Computer vision–ECCV 2014: 13th European conference, zurich, Switzerland, September 6-12, 2014, proce...
2014
-
[17]
Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. 2023. Improved Baselines with Visual Instruction Tuning
2023
-
[18]
Haotian Liu, Chunyuan Li, Yuheng Li, Bo Li, Yuanhan Zhang, Sheng Shen, and Yong Jae Lee. 2024. LLaVA-NeXT: Improved reasoning, OCR, and world knowl- edge. https://llava-vl.github.io/blog/2024-01-30-llava-next/
2024
-
[19]
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. 2023. Visual Instruc- tion Tuning
2023
-
[20]
Jian Liu, Yufeng Chen, and Jinan Xu. 2022. Multimedia event extraction from news with a unified contrastive learning framework. In Proceedings of the 30th ACM International Conference on Multimedia . 1945–1953
2022
-
[21]
Zheyuan Liu, Cristian Rodriguez-Opazo, Damien Teney, and Stephen Gould
-
[22]
Nguyen, Minh-Triet Tran, and Trung-Nghia Le
Hieu Nguyen, Phuc-Tan Nguyen, Thien-Phuc Tran, Minh-Quang Nguyen, Tam V. Nguyen, Minh-Triet Tran, and Trung-Nghia Le. 2025. OpenEvents V1: Large- Scale Benchmark Dataset for Multimodal Event Grounding. InACM International Conference on Multimedia
2025
-
[23]
Bryan A Plummer, Liwei Wang, Chris M Cervantes, Juan C Caicedo, Julia Hock- enmaier, and Svetlana Lazebnik. 2015. Flickr30k entities: Collecting region-to- phrase correspondences for richer image-to-sentence models. In Proceedings of the IEEE international conference on comput...
2015
-
[24]
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. 2021. Learning transferable visual models from natural language supervision. In International conference on machine learni...
2021
-
[25]
Hao Tan and Mohit Bansal. 2019. Lxmert: Learning cross-modality encoder representations from transformers. arXiv preprint arXiv:1908.07490 (2019)
2019 arXiv
-
[26]
Nguyen, Trong- Le Do, Duy-Nam Ly, Viet-Tham Huynh, Khanh-Duy Le, Mai-Khiem Tran, and Trung-Nghia Le
Thien-Phuc Tran, Minh-Quang Nguyen, Minh-Triet Tran, Tam V. Nguyen, Trong- Le Do, Duy-Nam Ly, Viet-Tham Huynh, Khanh-Duy Le, Mai-Khiem Tran, and Trung-Nghia Le. 2025. Event-Enriched Image Analysis Grand Challenge at ACM Multimedia 2025. In ACM International Conference on Multimedia
2025
-
[27]
Jianfeng Wang, Zhengyuan Yang, Xiaowei Hu, Linjie Li, Kevin Lin, Zhe Gan, Zicheng Liu, Ce Liu, and Lijuan Wang. 2022. Git: A generative image-to-text transformer for vision and language. arXiv preprint arXiv:2205.14100 (2022)
2022 arXiv
-
[28]
Liang Wang, Nan Yang, Xiaolong Huang, Linjun Yang, Rangan Majumder, and Furu Wei. 2024. Multilingual e5 text embeddings: A technical report. arXiv preprint arXiv:2402.05672 (2024)
2024 arXiv
-
[29]
Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, et al. 2024. Qwen2-vl: Enhancing vision-language model’s perception of the world at any resolution.arXiv preprint arXiv:2409.12191 (2024)
2024 arXiv
-
[30]
An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, et al. 2025. Qwen3 technical report. arXiv preprint arXiv:2505.09388 (2025)
2025 arXiv
-
[31]
Yuan Yao, Tianyu Yu, Ao Zhang, Chongyi Wang, Junbo Cui, Hongji Zhu, Tianchi Cai, Haoyu Li, Weilin Zhao, Zhihui He, et al. 2024. Minicpm-v: A gpt-4v level mllm on your phone. arXiv preprint arXiv:2408.01800 (2024)
2024 arXiv
-
[32]
Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, and Lucas Beyer. 2023. Sig- moid loss for language image pre-training. In Proceedings of the IEEE/CVF inter- national conference on computer vision . 11975–11986
2023
-
[33]
Xin Zhang, Yanzhao Zhang, Wen Xie, Mingxin Li, Ziqi Dai, Dingkun Long, Pengjun Xie, Meishan Zhang, Wenjie Li, and Min Zhang. 2024. GME: Im- proving Universal Multimodal Retrieval by Multimodal LLMs. arXiv preprint arXiv:2412.16855 (2024)
2024 arXiv
-
[34]
Xin Zhang, Yanzhao Zhang, Wen Xie, Mingxin Li, Ziqi Dai, Dingkun Long, Pengjun Xie, Meishan Zhang, Wenjie Li, and Min Zhang. 2025. Bridging Modali- ties: Improving Universal Multimodal Retrieval by Multimodal Large Language Models. In Proceedings of the Computer Vision and Pat...
2025
-
[35]
Yanzhao Zhang, Mingxin Li, Dingkun Long, Xin Zhang, Huan Lin, Baosong Yang, Pengjun Xie, An Yang, Dayiheng Liu, Junyang Lin, et al. 2025. Qwen3 Embedding: Advancing Text Embedding and Reranking Through Foundation Models. arXiv preprint arXiv:2506.05176 (2025)
2025 arXiv
-
[36]
Yucheng Zhou, Xiubo Geng, Tao Shen, Guodong Long, and Daxin Jiang. 2022. Eventbert: A pre-trained model for event correlation reasoning. In Proceedings of the ACM Web Conference 2022. 850–859
2022
-
[2021]
In Proceedings of the IEEE/CVF International Conference on Computer Vision
Image retrieval on real-life images with pre-trained vision-and-language models. In Proceedings of the IEEE/CVF International Conference on Computer Vision. 2125–2134
-
[2024]
M3-Embedding: Multi-Linguality, Multi-Functionality, Multi-Granularity Text Embeddings Through Self-Knowledge Distillation.Findings of the Association for Computational Linguistics: ACL (2024)
2024
Reviewed August 5, 2026 · model on record in the stance chip above.
Discussion (0). Sign in to comment.