Pith. sign in

REVIEW 4 major objections 4 minor 70 references

UniRank claims a single VLM can rerank hybrid text-image candidate sets in native form, and its two-stage training (instruction SFT then hard-negative mining with query-level GRPO) yields 8.9% and 7.3% Recall@1 gains over SOTA baselines.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-03 03:25 UTC pith:YLJOXH7S

load-bearing objection A well-built reranking pipeline with a real design idea, but the patent-domain gain rests on unvalidated Gemini labels and there's no significance testing. the 4 major comments →

arxiv 2603.29897 v2 pith:YLJOXH7S submitted 2026-02-08 cs.IR cs.AI

UniRank: End-to-End Domain-Specific Reranking of Hybrid Text-Image Candidates

Yupei Yang , Lin Yang , Wanxi Deng , Lin Qu , Shikui Tu , Lei Xu This is my paper
classification cs.IR cs.AI
keywords multimodal rerankinghybrid text-image candidatesvision-language modelsdomain-specific fine-tuninghard-negative miningreinforcement learning from human feedbackGRPOquery-level grouping
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

UniRank claims that a vision-language model can act as a reranker over candidate sets that mix text and images, scoring every candidate in its native modality instead of converting one kind into the other. The core mechanism is a yes/no relevance judgment whose label-token logits are converted into a scalar score, making text, image, and mixed pairs comparable on the same scale. The paper proposes a two-stage domain-adaptation pipeline: instruction-driven supervised fine-tuning to calibrate cross-modal scoring, then hard-negative mining from the fine-tuned model's top-ranked mistakes followed by query-level GRPO reinforcement learning to sharpen ordering. The authors report consistent gains over strong text-only and VLM-based rerankers, with Recall@1 up 8.9% on scientific literature layout retrieval and 7.3% on design patent search, and lower storage and latency than image-homogenized alternatives.

Core claim

The central discovery is that the modality gap in hybrid reranking need not be closed by forcing all inputs into one modality. The authors show that a VLM prompted to answer 'yes' or 'no' for relevance yields a usable scalar score from the logit difference s=σ(logit_yes − logit_no), and that this score is sufficiently well-calibrated across text, image, and mixed query–candidate pairs to support top-k reranking. Building on that interface, they demonstrate that end-to-end domain adaptation is possible: SFT establishes calibrated scoring, then hard negatives—non-relevant candidates the SFT model mistakenly scores highly—are mined to form preference pairs, and GRPO with query-level groups opti

What carries the argument

The load-bearing object is the label-token scoring function s(q,c)=σ(logit(yes)−logit(no)) extracted from a VLM's constrained yes/no output, which provides a single scalar score across text, image, and cross-modal pairs. The second essential component is query-level GRPO: preference groups consist of all candidate prompts sharing one query, so the policy gradient directly optimizes listwise ordering rather than per-prompt label accuracy. Hard-negative mining—selecting the SFT reranker's top-N mistakes—supplies the training signal that makes the preference data informative.

Load-bearing premise

The patent-domain results treat binary risk labels produced by an off-the-shelf API model as ground truth for 'substantially similar in overall visual impression'; if those labels diverge from human legal judgment, the measured 7.3-point patent gain is a gain at predicting the API model, not at reranking for infringement risk.

What would settle it

Take the design-patent test set and have human annotators independently label a sample of query–patent pairs as high-risk/low-risk, then recompute Recall@1 and NDCG for UniRank and the baselines against those human labels; if UniRank's 7.3% advantage shrinks or reverses, the patent-domain claim is an artifact of the synthetic labeler. A cheaper first check is a small inter-annotator agreement study between the API model and human judges.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Domain-specific multimodal rerankers can be trained end-to-end from a general VLM using only relevance labels for SFT and automatically mined hard negatives for preference alignment.
  • Reranking pipelines can keep candidates in native form, eliminating the storage and latency costs of converting text to images.
  • The query-level GRPO formulation, contrasting candidates under the same query, is a general recipe for turning a pointwise generative scorer into a listwise ranker.
  • The authors' ablations show the full pipeline, not any single stage, is responsible for the gains; SFT alone helps, hard negatives help more, and RLHF adds the rest.
  • The method transfers across first-stage retrievers and backbone sizes, so the training recipe, not a tuned candidate distribution, drives improvement.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • A natural extension is to multi-grade relevance: the same logit-difference interface could output a distribution over more than two labels, giving graded scores without retraining the interface.
  • Because design-patent supervision is generated by an API model, the 7.3% gain could partly reflect predicting that model's visual-similarity notion; a human-annotated subset would separate true domain improvement from labeler alignment.
  • The storage/latency argument suggests the approach is a candidate for on-device retrieval, where image-converted text is prohibitive; measuring performance on mobile or embedded hardware would test that claim.
  • Query-level grouping could be applied to other pointwise-to-listwise tasks (e.g., recommendation reranking or fact verification) by grouping candidates sharing the same context.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The paper proposes UniRank, a VLM-based reranking framework for hybrid text-image candidate sets. The method avoids modality conversion by feeding queries and candidates in their native modalities to a VLM and deriving a scalar score from label-token logits (Eq. 4). Training is a two-stage pipeline: instruction-driven SFT on in-domain relevance judgments, followed by hard-negative mining, reward-model training, and query-level GRPO. The authors evaluate on MMDocIR academic-paper layout retrieval and a design-patent search dataset, reporting consistent gains over text-only and VLM rerankers, with headline Recall@1 improvements of 8.9% and 7.3%, and also report efficiency advantages from native hybrid scoring.

Significance. If the results are taken at face value, the paper makes a useful practical contribution: it shows that a general VLM can be adapted end-to-end to domain-specific hybrid reranking without image-ifying text, and it provides a recipe (SFT + hard-negative preference construction + query-level GRPO) that is plausibly transferable. The manuscript has several strengths: it releases code; train/hold/test splits are disjoint; ablations isolate SFT, hard-negative mining, RLHF, and query-level grouping; robustness is tested across five first-stage retrievers and three backbone sizes; and efficiency is measured rather than asserted. However, the patent-domain evaluation rests entirely on API-model-generated labels with no human validation, and the reported metrics lack variance estimates or significance testing. These issues materially limit confidence in the general claim that UniRank improves real-world hybrid reranking, especially in the patent domain.

major comments (4)
  1. [§4.1 / Appendix B.2] The design-patent ground truth is produced by gemini-3-pro-preview and 'treated as supervision for training and evaluation' (§4.1). Appendix B.2 describes the labeling procedure but reports no human validation, inter-annotator agreement, or comparison with legal/common-observer judgments. Because UniRank is SFT/RLHF-tuned on these Gemini labels and evaluated on separate draws from the same labeling process, the 7.3% Recall@1 gain over zero-shot general VLMs may simply reflect how well UniRank internalizes Gemini's decision boundary. This is not equation-level circularity, but the evaluation target is the API model's opinion. The patent-domain claim needs either a human-validated test subset or a reframing of the benchmark as 'predicting Gemini risk labels.'
  2. [Appendix A, Eq. (17)] The metric called Recall@k is defined as I[rank_q <= k], i.e., whether at least one relevant candidate appears in the top k. This is Success@k / Hit@k, not standard recall (fraction of relevant items retrieved). Since Recall@1 is a headline claim in the abstract, the mislabeling is consequential. The authors should rename the metric or additionally report standard recall@k; otherwise readers may over-interpret the numbers.
  3. [Tables 2, 3, 5, 6, 7] No variance, confidence intervals, or significance tests are reported anywhere. Several key comparisons are close (e.g., patent Recall@1: UniRank 65.2 vs Qwen3-VL-Reranker 57.9 in Table 3; patent NDCG@3 differences in Table 7 are under 1 point). The claim that UniRank 'consistently outperforms' baselines is therefore not statistically supported. I request repeated runs across seeds, or at least significance/paired tests on the existing query-level scores, for the main tables.
  4. [§4.3, Table 7] The 'w/o SFT' ablation is underspecified. Hard-negative mining (§3.3.1) is defined using the SFT reranker's scores; if SFT is skipped, it is unclear what model mines the hard negatives and how the preference data are constructed in this variant. Without this detail, the ablation may conflate removal of SFT with a change in preference-data quality. Please specify the exact protocol used for the 'w/o SFT' row.
minor comments (4)
  1. [Appendix C, Tables 8–9] Hyperparameters such as hard-negative pool size N, GRPO group size G, LoRA rank/alpha, and KL regularization are fixed without sensitivity analysis. Since these are free choices in the pipeline, a short sensitivity study (or at least a discussion) would strengthen the recipe's credibility.
  2. [Table 4] The storage comparison mixes representation formats: MM-R5's 83.5 GB reflects image-encoding of all scientific candidates, while UniRank stores native text plus images. Please clarify the exact candidate corpus, image resolution, and encoding assumptions so the 6.9x/1.4x storage ratios are reproducible.
  3. [Table 6] In the Patent block, the UniRank-2B Recall@1 entry appears as '62.285.391.8' — likely a spacing/formatting error. Please fix the table rendering.
  4. [Section 3.3 / Title] The pipeline is called RLHF, but preference data are constructed automatically from hard-negative mining and labels, not from human annotators. Consider calling it 'preference-based RL' or explicitly noting that human feedback is not involved, to avoid terminological overreach.

Circularity Check

0 steps flagged

No significant circularity: UniRank's scoring interface, training stages, and evaluation splits are independently defined; the Gemini-generated patent labels are a data-validity caveat, not a circular derivation.

full rationale

UniRank's derivation chain is not circular. The scoring function in Eq. (4), s = sigmoid(logit_yes - logit_no), is a stated interface rather than a parameter fitted to the benchmark; the SFT objective in Eq. (7) trains label-token logits on DSFT, with disjoint Dhold and Dtest splits. Hard-negative mining uses the SFT model's own scores only to select candidate prompts, while the preference labels come from ground-truth relevance labels (MMDocIR or Gemini risk labels), not from the SFT model's predictions. The reward model (Eq. 12) and GRPO objective (Eq. 15) are trained on those preference data, and the reported Recall@k/NDCG@k/MRR numbers are computed on a disjoint test set, so no reported metric reduces by construction to a fitted parameter. The only notable caveat is that the patent-domain labels are produced by gemini-3-pro-preview and 'treated as supervision for training and evaluation' (Section 4.1; Appendix B.2), with no human validation or inter-annotator agreement reported. This is a construct-validity / data-quality concern: UniRank may be learning to mimic one API model's decision boundary rather than genuine human 'substantially similar' judgments, which could overstate the practical patent-domain gains. But it is not formal circularity, because the test labels are separate draws and the comparison against baselines is empirically contingent, not forced by the paper's equations. No load-bearing self-citations, imported uniqueness theorems, or ansatz-smuggling citations were found. The paper is self-contained against external benchmarks and explicit baselines.

Axiom & Free-Parameter Ledger

4 free parameters · 4 axioms · 0 invented entities

The paper introduces no new scientific constants or entities. It relies on manually chosen training hyperparameters and, more importantly, on two dataset assumptions (MMDocIR labels and Gemini-generated patent labels) whose validity is untested. The most load-bearing choice is that Gemini labels are acceptable ground truth.

free parameters (4)
  • hard-negative pool size N = 30 (scientific), 5 (patent)
    Chosen by hand per domain (Appendix C.1); determines the difficulty and composition of preference data.
  • GRPO group size G = 30 (scientific), 5 (patent)
    Query-level group size used in Eqs. (13)-(15); set per domain, not derived.
  • LoRA rank and alpha = r=16, alpha=32
    Standard PEFT hyperparameters listed in Table 8; not fitted to test data but manually selected.
  • KL regularization coefficient = 0.04
    GRPO hyperparameter from Table 9; chosen by hand for stability of RLHF.
axioms (4)
  • domain assumption gemini-3-pro-preview risk labels are valid ground truth for design-patent substantial similarity.
    Appendix B.2: labels are generated by an API model and treated as supervision for both training and evaluation; no human validation or inter-annotator agreement is reported.
  • domain assumption MMDocIR layout-level relevance labels are trustworthy ground truth for the scientific domain.
    Appendix B.1: the paper adopts the MMDocIR Academic Paper layout-retrieval labels as supervision and evaluation without additional verification.
  • domain assumption The sigmoid of the label-token logit difference in Eq. (4) yields scores that are comparable across text, image, and text-image query-candidate pairs.
    Section 3.2.1 introduces this scoring interface; cross-modal comparability of logit differences is assumed, not proven.
  • domain assumption Binary relevance and the metrics Recall@k, NDCG@k, and MRR adequately capture reranking quality in these domains.
    Appendix A defines metrics with binary labels; the paper does not justify that binary relevance is sufficient for the patent risk-screening task.

pith-pipeline@v1.3.0-alltime-deepseek · 21700 in / 13269 out tokens · 135283 ms · 2026-08-03T03:25:16.521163+00:00 · methodology

0 comments
read the original abstract

Reranking is a critical component in many information retrieval pipelines. Despite remarkable progress in text-only settings, multimodal reranking remains challenging, particularly when the candidate set contains hybrid text and image items. A key difficulty is the modality gap: a text reranker is intrinsically closer to text candidates than to image candidates, leading to biased and suboptimal cross-modal ranking. Vision-language models (VLMs) mitigate this gap through strong cross-modal alignment and have recently been adopted to build multimodal rerankers. However, most VLM-based rerankers encode all candidates as images, and treating text as images introduces substantial computational overhead. Meanwhile, existing open-source multimodal rerankers are typically trained on general-domain data and often underperform in domain-specific scenarios. To address these limitations, we propose UniRank, a VLM-based reranking framework that natively scores and orders hybrid text-image candidates without any modality conversion. Building on this hybrid scoring interface, UniRank provides an end-to-end domain adaptation pipeline that includes: (1) an instruction-tuning stage that learns calibrated cross-modal relevance scoring by mapping label-token likelihoods to a unified scalar score; and (2) a hard-negative-driven preference alignment stage that constructs in-domain pairwise preferences and performs query-level policy optimization through reinforcement learning from human feedback (RLHF). Extensive experiments on scientific literature retrieval and design patent search demonstrate that UniRank consistently outperforms state-of-the-art baselines, improving Recall@1 by 8.9% and 7.3%, respectively.

Figures

Figures reproduced from arXiv: 2603.29897 by Lei Xu, Lin Qu, Lin Yang, Shikui Tu, Wanxi Deng, Yupei Yang.

Figure 1
Figure 1. Figure 1: Overview of [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Example from MMDocIR Academic Paper (layout-retrieval). Given a text-only question, the retriever/reranker selects [PITH_FULL_IMAGE:figures/full_fig_p012_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Example from the design patent search task. Given a target product as the query (image + text), the reranker scores [PITH_FULL_IMAGE:figures/full_fig_p012_3.png] view at source ↗
Figure 5
Figure 5. Figure 5: Illustrative SFT and preference-data examples for [PITH_FULL_IMAGE:figures/full_fig_p014_5.png] view at source ↗
Figure 4
Figure 4. Figure 4: Illustrative SFT and preference-data examples for [PITH_FULL_IMAGE:figures/full_fig_p014_4.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

70 extracted references · 28 linked inside Pith

  1. [1]

    1999.Modern information retrieval

    Ricardo Baeza-Yates, Berthier Ribeiro-Neto, et al . 1999.Modern information retrieval. Vol. 463. ACM press New York

    1999

  2. [2]

    Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. 2023. Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyond.arXiv preprint arXiv:2308.12966(2023)

    Pith/arXiv arXiv 2023

  3. [3]

    Shuai Bai, Yuxuan Cai, Ruizhe Chen, Keqin Chen, Xionghui Chen, Zesen Cheng, Lianghao Deng, Wei Ding, Chang Gao, Chunjiang Ge, Wenbin Ge, Zhifang Guo, Qidong Huang, Jie Huang, Fei Huang, Binyuan Hui, Shutong Jiang, Zhaohai Li, Mingsheng Li, Mei Li, Kaixin Li, Zicheng Lin, Junyang Lin, Xuejing Liu, Jiawei Liu, Chenglong Liu, Yang Liu, Dayiheng Liu, Shixuan ...

    Pith/arXiv arXiv 2025

  4. [4]

    Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, Humen Zhong, Yuanzhi Zhu, Mingkun Yang, Zhaohai Li, Jianqiang Wan, Pengfei Wang, Wei Ding, Zheren Fu, Yiheng Xu, Jiabo Ye, Xi Zhang, Tianbao Xie, Zesen Cheng, Hang Zhang, Zhibo Yang, Haiyang Xu, and Junyang Lin. 2025. Qwen2.5-VL Technical Rep...

    Pith/arXiv arXiv 2025

  5. [5]

    Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, et al. 2022. Training a helpful and harmless assistant with reinforcement learning from human feedback.arXiv preprint arXiv:2204.05862(2022)

    Pith/arXiv arXiv 2022

  6. [6]

    Ralph Allan Bradley and Milton E Terry. 1952. Rank analysis of incomplete block designs: I. the method of paired comparisons.Biometrika39, 3/4 (1952), 324–345

    1952

  7. [7]

    Jianlv Chen, Shitao Xiao, Peitian Zhang, Kun Luo, Defu Lian, and Zheng Liu

  8. [8]

    Zhanpeng Chen, Chengjin Xu, Yiyan Qi, and Jian Guo. 2024. Mllm is a strong reranker: Advancing multimodal retrieval-augmented generation via knowledge- enhanced reranking and noise-injected training.arXiv preprint arXiv:2407.21439 (2024)

    Pith/arXiv arXiv 2024

  9. [9]

    2010.Introduction to modern information retrieval

    Gobinda G Chowdhury. 2010.Introduction to modern information retrieval. Facet publishing

    2010

  10. [10]

    Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. 2023. Qlora: Efficient finetuning of quantized llms.Advances in neural information processing systems36 (2023), 10088–10115

    2023

  11. [11]

    Ning Ding, Yujia Qin, Guang Yang, Fuchao Wei, Zonghan Yang, Yusheng Su, Shengding Hu, Yulin Chen, Chi-Min Chan, Weize Chen, et al. 2023. Parameter- efficient fine-tuning of large-scale pre-trained language models.Nature machine intelligence5, 3 (2023), 220–235

    2023

  12. [12]

    Kuicai Dong, Yujing Chang, Xin Deik Goh, Dexun Li, Ruiming Tang, and Yong Liu. 2025. Mmdocir: Benchmarking multi-modal retrieval for long documents. arXiv preprint arXiv:2501.08828(2025)

    arXiv 2025

  13. [13]

    Alexey Dosovitskiy. 2020. An image is worth 16x16 words: Transformers for image recognition at scale.arXiv preprint arXiv:2010.11929(2020)

    Pith/arXiv arXiv 2020

  14. [14]

    Darren Edge, Ha Trinh, Newman Cheng, Joshua Bradley, Alex Chao, Apurva Mody, Steven Truitt, Dasha Metropolitansky, Robert Osazuwa Ness, and Jonathan Larson. 2024. From local to global: A graph rag approach to query-focused summarization.arXiv preprint arXiv:2404.16130(2024)

    Pith/arXiv arXiv 2024

  15. [15]

    Wenqi Fan, Yujuan Ding, Liangbo Ning, Shijie Wang, Hengyun Li, Dawei Yin, Tat-Seng Chua, and Qing Li. 2024. A survey on rag meeting llms: Towards retrieval-augmented large language models. InProceedings of the 30th ACM SIGKDD conference on knowledge discovery and data mining. 6491–6501

    2024

  16. [16]

    Manuel Faysse, Hugues Sibille, Tony Wu, Bilel Omrani, Gautier Viaud, Céline Hudelot, and Pierre Colombo. 2024. Colpali: Efficient document retrieval with vision language models.arXiv preprint arXiv:2407.01449(2024)

    Pith/arXiv arXiv 2024

  17. [17]

    Yunfan Gao, Yun Xiong, Xinyu Gao, Kangxiang Jia, Jinliu Pan, Yuxi Bi, Yixin Dai, Jiawei Sun, Haofen Wang, and Haofen Wang. 2023. Retrieval-augmented generation for large language models: A survey.arXiv preprint arXiv:2312.10997 2, 1 (2023)

    Pith/arXiv arXiv 2023

  18. [18]

    Gregor Geigle, Jonas Pfeiffer, Nils Reimers, Ivan Vulić, and Iryna Gurevych. 2022. Retrieve fast, rerank smart: Cooperative and joint approaches for improved cross- modal retrieval.Transactions of the Association for Computational Linguistics10 (2022), 503–521

    2022

  19. [19]

    Petros Stylianos Giouroukis, Dimitris Dimitriadis, Dimitrios Papadopoulos, Zhen- wen Shao, and Grigorios Tsoumakas. 2025. What’s the Best Way to Retrieve Slides? A Comparative Study of Multimodal, Caption-Based, and Hybrid Retrieval Techniques.arXiv preprint arXiv:2509.15211(2025)

    arXiv 2025

  20. [20]

    Kailash A Hambarde and Hugo Proenca. 2023. Information retrieval: recent advances and beyond.IEEE Access11 (2023), 76581–76604

    2023

  21. [21]

    Zeyu Han, Chao Gao, Jinyang Liu, Jeff Zhang, and Sai Qian Zhang. 2024. Parameter-efficient fine-tuning for large models: A comprehensive survey.arXiv preprint arXiv:2403.14608(2024)

    Pith/arXiv arXiv 2024

  22. [22]

    Mariya Hendriksen. 2022. Multimodal retrieval in e-commerce: From categories to images, text, and back. InEuropean Conference on Information Retrieval. Springer, 505–512

    2022

  23. [23]

    Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen, et al. 2022. Lora: Low-rank adaptation of large language models.ICLR1, 2 (2022), 3

    2022

  24. [24]

    Cheng Jiang, Pengle Zhang, Ying Ni, Xiaoli Wang, Hanghang Peng, Sen Liu, Mengdi Fei, Yuxin He, Yaxuan Xiao, Jin Huang, et al. 2025. Multimodal retrieval- augmented generation for financial documents: image-centric analysis of charts and tables with large language models.The Visual Computer(2025), 1–14

    2025

  25. [25]

    Vladimir Karpukhin, Barlas Oguz, Sewon Min, Patrick SH Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih. 2020. Dense Passage Retrieval for Open-Domain Question Answering.. InEMNLP (1). 6769–6781

    2020

  26. [26]

    Omar Khattab and Matei Zaharia. 2020. Colbert: Efficient and effective passage search via contextualized late interaction over bert. InProceedings of the 43rd KDD ’26, August 09–13, 2026, Jeju Island, Republic of Korea Yupei Yang et al. International ACM SIGIR conference on research and development in Information Retrieval. 39–48

    2020

  27. [27]

    Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, et al. 2020. Retrieval-augmented generation for knowledge-intensive nlp tasks. Advances in neural information processing systems33 (2020), 9459–9474

    2020

  28. [28]

    Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. 2023. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. InInternational conference on machine learning. PMLR, 19730–19742

    2023

  29. [29]

    Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. 2022. Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation. InInternational conference on machine learning. PMLR, 12888–12900

    2022

  30. [30]

    Mingxin Li, Yanzhao Zhang, Dingkun Long, Keqin Chen, Sibo Song, Shuai Bai, Zhibo Yang, Pengjun Xie, An Yang, Dayiheng Liu, et al . 2026. Qwen3-VL- Embedding and Qwen3-VL-Reranker: A Unified Framework for State-of-the-Art Multimodal Retrieval and Ranking.arXiv preprint arXiv:2601.04720(2026)

    Pith/arXiv arXiv 2026

  31. [31]

    Victor Weixin Liang, Yuhui Zhang, Yongchan Kwon, Serena Yeung, and James Y Zou. 2022. Mind the gap: Understanding the modality gap in multi-modal con- trastive representation learning.Advances in Neural Information Processing Systems35 (2022), 17612–17625

    2022

  32. [32]

    Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. 2023. Visual in- struction tuning.Advances in neural information processing systems36 (2023), 34892–34916

    2023

  33. [33]

    Zhenghao Liu, Xingsheng Zhu, Tianshuo Zhou, Xinyi Zhang, Xiaoyuan Yi, Yukun Yan, Ge Yu, and Maosong Sun. 2025. Benchmarking retrieval-augmented gen- eration in multi-modal contexts. InProceedings of the 33rd ACM International Conference on Multimedia. 4817–4826

    2025

  34. [34]

    Xueguang Ma, Sheng-Chieh Lin, Minghan Li, Wenhu Chen, and Jimmy Lin

  35. [35]

    2008.Introduction to information retrieval

    Christopher D Manning. 2008.Introduction to information retrieval. Syngress Publishing,

    2008

  36. [36]

    Unifying multimodal retrieval via document screenshot embedding.arXiv preprint arXiv:2406.11251(2024)

    Pith/arXiv arXiv 2024

  37. [37]

    Matin Mortaheb, Mohammad A Amir Khojastepour, Srimat T Chakradhar, and Sennur Ulukus. 2025. Re-ranking the context for multimodal retrieval augmented generation.arXiv preprint arXiv:2501.04695(2025)

    Pith/arXiv arXiv 2025

  38. [38]

    Lang Mei, Siyu Mo, Zhihan Yang, and Chong Chen. 2025. A survey of multimodal retrieval-augmented generation.arXiv preprint arXiv:2504.08748(2025)

    Pith/arXiv arXiv 2025

  39. [39]

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. 2021. Learning transferable visual models from natural language supervision. InInternational conference on machine learning. PmLR, 8748–8763

    2021

  40. [40]

    Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. 2022. Training language models to follow instructions with human feedback.Advances in neural information processing systems35 (2022), 27730–27744

    2022

  41. [41]

    John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov

  42. [42]

    Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. 2023. Direct preference optimization: Your language model is secretly a reward model.Advances in neural information processing systems36 (2023), 53728–53741

    2023

  43. [43]

    Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Yang Wu, et al. 2024. Deepseekmath: Pushing the limits of mathematical reasoning in open language models.arXiv preprint arXiv:2402.03300(2024)

    Pith/arXiv arXiv 2024

  44. [44]

    Zhiqing Sun, Sheng Shen, Shengcao Cao, Haotian Liu, Chunyuan Li, Yikang Shen, Chuang Gan, Liangyan Gui, Yu-Xiong Wang, Yiming Yang, et al . 2024. Aligning large multimodal models with factually augmented rlhf. InFindings of the Association for Computational Linguistics: ACL 2024. 13088–13110

    2024

  45. [45]

    A Shakir, D Koenig, J Lipp, and S Lee. 2024. Boost your search with the crispy mixedbread rerank models

    2024

  46. [46]

    Feng Wang, Yuqing Li, and Han Xiao. 2025. Jina-reranker-v3: Last but Not Late Interaction for Listwise Document Reranking.arXiv preprint arXiv:2509.25085 (2025)

    arXiv 2025

  47. [47]

    Meng Wang, Hao Li, Dacheng Tao, Ke Lu, and Xindong Wu. 2012. Multimodal graph-based reranking for web image search.IEEE transactions on image process- ing21, 11 (2012), 4649–4661

    2012

  48. [48]

    Gemini Team, Rohan Anil, Sebastian Borgeaud, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, Katie Millican, et al. 2023. Gemini: a family of highly capable multimodal models.arXiv preprint arXiv:2312.11805(2023)

    Pith/arXiv arXiv 2023

  49. [49]

    Xiao Wang, Craig Macdonald, Nicola Tonellotto, and Iadh Ounis. 2023. ColBERT- PRF: Semantic pseudo-relevance feedback for dense passage and document re- trieval.ACM Transactions on the Web17, 1 (2023), 1–39

    2023

  50. [50]

    Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Alisa Liu, Noah A Smith, Daniel Khashabi, and Hannaneh Hajishirzi. 2023. Self-instruct: Aligning language mod- els with self-generated instructions. InProceedings of the 61st annual meeting of the association for computational linguistics (volume 1: long papers). 13484–13508

    2023

  51. [51]

    Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Yang Fan, Kai Dang, Mengfei Du, Xuancheng Ren, Rui Men, Dayiheng Liu, Chang Zhou, Jingren Zhou, and Junyang Lin. 2024. Qwen2-VL: Enhancing Vision-Language Model’s Perception of the World at Any Resolution.arXiv preprint arXiv:2409.12191(2024)

    Pith/arXiv arXiv 2024

  52. [52]

    Haoyang Wen, Honglei Zhuang, Hamed Zamani, Alexander Hauptmann, and Michael Bendersky. 2024. Multimodal reranking for knowledge-intensive visual question answering.arXiv preprint arXiv:2407.12277(2024)

    Pith/arXiv arXiv 2024

  53. [53]

    Siwei Wu, Yizhi Li, Kang Zhu, Ge Zhang, Yiming Liang, Kaijing Ma, Chenghao Xiao, Haoran Zhang, Bohao Yang, Wenhu Chen, et al. 2024. Scimmir: Benchmark- ing scientific multi-modal information retrieval. InFindings of the Association for Computational Linguistics: ACL 2024. 12560–12574

    2024

  54. [54]

    Jason Wei, Maarten Bosma, Vincent Y Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M Dai, and Quoc V Le. 2021. Finetuned language models are zero-shot learners.arXiv preprint arXiv:2109.01652(2021)

    Pith/arXiv arXiv 2021

  55. [55]

    Enqiang Xu, Xinhui Li, Zhigong Zhou, Jiahao Ji, Jinyuan Zhao, Dadong Miao, Songlin Wang, Lin Liu, and Sulong Xu. 2024. Advancing Re-Ranking with Multimodal Fusion and Target-Oriented Auxiliary Tasks in E-Commerce Search. InProceedings of the 33rd ACM International Conference on Information and Knowledge Management. 5007–5014

    2024

  56. [56]

    Mingjun Xu, Jinhan Dong, Jue Hou, Zehui Wang, Sihang Li, Zhifeng Gao, Renxin Zhong, and Hengxing Cai. 2025. MM-R5: MultiModal Reasoning-Enhanced ReRanker via Reinforcement Learning for Document Retrieval.arXiv preprint arXiv:2506.12364(2025)

    Pith/arXiv arXiv 2025

  57. [57]

    Shitao Xiao, Zheng Liu, Peitian Zhang, and Niklas Muennighoff. 2023. C-Pack: Packaged Resources To Advance General Chinese Embedding. arXiv:2309.07597 [cs.CL]

    Pith/arXiv arXiv 2023

  58. [58]

    Jing Yao, JunLin Li, and Beilei Zhao. 2024. Research on Military Multimodal Knowledge Fusion Representation Based on Retrieval-Augmented Generation. In 2024 5th International Conference on Computer, Big Data and Artificial Intelligence (ICCBD+ AI). IEEE, 221–228

    2024

  59. [59]

    Zhenrui Yue, Huimin Zeng, Yueqi Wang, Julian McAuley, and Dong Wang. 2025. Preference-Optimized Retrieval and Ranking for Efficient Multimodal Recom- mendation. InProceedings of the 31st ACM SIGKDD Conference on Knowledge Discovery and Data Mining V. 2. 3692–3703

    2025

  60. [60]

    Xiukang Yang, Jingguo Ge, Hui Li, Liangxiong Li, and Bingzhen Wu. 2025. ReID: Re-ranking through image description for object re-identification.Pattern Recog- nition(2025), 112552

    2025

  61. [61]

    Yanzhao Zhang, Mingxin Li, Dingkun Long, Xin Zhang, Huan Lin, Baosong Yang, Pengjun Xie, An Yang, Dayiheng Liu, Junyang Lin, et al. 2025. Qwen3 Embedding: Advancing Text Embedding and Reranking Through Foundation Models.arXiv preprint arXiv:2506.05176(2025)

    Pith/arXiv arXiv 2025

  62. [62]

    Ruochen Zhao, Hailin Chen, Weishi Wang, Fangkai Jiao, Xuan Long Do, Chengwei Qin, Bosheng Ding, Xiaobao Guo, Minzhi Li, Xingxuan Li, et al. 2023. Retrieving multimodal information for augmented generation: A survey.arXiv preprint arXiv:2303.10868(2023)

    Pith/arXiv arXiv 2023

  63. [63]

    Qingru Zhang, Minshuo Chen, Alexander Bukharin, Nikos Karampatziakis, Pengcheng He, Yu Cheng, Weizhu Chen, and Tuo Zhao. 2023. Adalora: Adaptive budget allocation for parameter-efficient fine-tuning.arXiv preprint arXiv:2303.10512(2023)

    Pith/arXiv arXiv 2023

  64. [64]

    Jeffrey Zhou, Tianjian Lu, Swaroop Mishra, Siddhartha Brahma, Sujoy Basu, Yi Luan, Denny Zhou, and Le Hou. 2023. Instruction-following evaluation for large language models.arXiv preprint arXiv:2311.07911(2023)

    Pith/arXiv arXiv 2023

  65. [65]

    Yinxin Zhou, Qin Luo, Bin Feng, and Bang Wang. 2025. Large Language Models for Reranking: A Survey.Authorea Preprints(2025)

    2025

  66. [66]

    Yuze Zhao, Jintao Huang, Jinghan Hu, Xingjun Wang, Yunlin Mao, Daoze Zhang, Zeyinzi Jiang, Zhikai Wu, Baole Ai, Ang Wang, et al . 2025. Swift: a scalable lightweight infrastructure for fine-tuning. InProceedings of the AAAI Conference on Artificial Intelligence, Vol. 39. 29733–29735

    2025

  67. [67]

    Yutao Zhu, Huaying Yuan, Shuting Wang, Jiongnan Liu, Wenhan Liu, Chenlong Deng, Haonan Chen, Zheng Liu, Zhicheng Dou, and Ji-Rong Wen. 2025. Large language models for information retrieval: A survey.ACM Transactions on Information Systems44, 1 (2025), 1–54. UniRank: End-to-End Domain-Specific Reranking of Hybrid Text-Image Candidates KDD ’26, August 09–13...

    2025

  68. [69]

    Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny. 2023. Minigpt-4: Enhancing vision-language understanding with advanced large lan- guage models.arXiv preprint arXiv:2304.10592(2023)

    Pith/arXiv arXiv 2023

  69. [2017]

    Proximal policy optimization algorithms.arXiv preprint arXiv:1707.06347 (2017)

    Pith/arXiv arXiv 2017

  70. [2024]

    arXiv:2402.03216 [cs.CL]

    BGE M3-Embedding: Multi-Lingual, Multi-Functionality, Multi-Granularity Text Embeddings Through Self-Knowledge Distillation. arXiv:2402.03216 [cs.CL]

    Pith/arXiv arXiv