REVIEW 5 major objections 6 minor 29 references
Who is in the Spotlight: The Hidden Bias Undermining Multimodal Retrieval-Augmented Generation
T0 review · 5 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash
Pith's one-line read Retrieval-augmented generation answers drop when the correct evidence sits in the middle of a retrieved list, and the effect worsens logarithmically as the list grows.
desk verdict Useful qualitative finding on position bias in multimodal RAG, but the quantitative scaling law is not reproducible from the paper's own table and several headline claims overreach the data. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing object is the Position Sensitivity Index, $PSI_p = \frac{1}{p}\sum_{i\in T_p} A_i - \frac{1}{p}\sum_{j\in B_p} A_j$, which is the gap between the average accuracy of the $p$ highest-scoring and $p$ lowest-scoring positions; with $p=1$ it contrasts the best position (usually top or end) with the worst (usually middle). The experiments isolate position from content by permuting gold evidence among distractors with ten randomized runs per configuration. For the mechanism, the paper extracts cross-modal attention weights from the 14th decoder layer and measures concentration with a normalized two-dimensional sparsity index, $S_{2D} = 1 - N2E$, showing that middle layers concentrate attention most sharply.
What would settle it
Re-run the position-perturbation protocol on text-only and mixed-modality benchmarks at $k$ values of 30 and 40 and check whether $PSI_p$ still falls on a straight line when plotted against $\ln(k)$ with slope near 0.030-0.040; if the relationship flattens or bends, the log law is refuted. Alternatively, a vision-language model with a different cross-modal fusion design that shows flat $PSI_p$ across $k$ would refute the claim that multimodal interactions intrinsically amplify position bias.
Extended reading notes
Core claim
The central claim is that position bias in multimodal RAG is systematic, quantifiable, and larger than in unimodal RAG. With content held fixed, placing the single correct evidence document at the top or end of the retrieved sequence yields higher answer accuracy than placing it in the middle, producing a U-shaped accuracy curve that appears consistently in text, image, and interleaved image-text tasks. The bias is stronger when modalities interact than in text-only settings, and it increases with retrieval count $k$ according to $PSI_p = \alpha \ln(k) + \beta$, with fitted slopes between about 0.030 and 0.040. Attention visualizations at one middle decoder layer show that top-positioned evidence captures stronger cross-modal attention while middle evidence is diluted, which the paper offers as the mechanism behind the accuracy gap.
Load-bearing premise
The logarithmic growth law $PSI_p = \alpha \ln(k) + \beta$ is fitted only on the ChartQA image-only setting for retrieval sizes up to 20, yet it is stated as a general law for text and mixed-modality RAG as well.
Editorial extensions
If this is right
- Middle-position evidence reliably underperforms top and end positions in text-only, image-only, and mixed-modality RAG settings.
- Multimodal interactions increase position sensitivity relative to text-only for most models, so mixed-evidence tasks inherit a stronger ordering bias.
- Retrieval count drives the bias: $PSI_p$ grows with $k$ following $PSI_p = \alpha \ln(k) + \beta$, implying that each doubling of the retrieved list adds a roughly constant amount of positional bias.
- Attention analysis localizes the effect: cross-modal attention at layer 14 favors top-positioned evidence and dilutes middle evidence.
- RAG pipelines should either limit retrieval range or apply position-aware reweighting or reordering to prevent accuracy from declining as evidence lists lengthen.
Reading between the lines
- Beyond the paper's fitted range, if the log-linear law continues past $k=20$, then beyond some list size additional retrieved documents will hurt reliability more than they help; the fitted slope gives a concrete way to estimate that crossover.
- Editorial inference: the U-shape resembles the 'lost in the middle' effect reported for long-context language models; an experiment comparing long text-only lists with multimodal lists at the same $k$ would show whether multimodal fusion adds a new mechanism or merely inherits the decoder's positional prior.
- Editorial inference: since attention sparsity peaks in middle decoder layers, a targeted intervention at those layers (such as attention reweighting) is a testable debiasing strategy that the paper does not itself evaluate.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper studies position bias in multimodal retrieval-augmented generation (RAG). It introduces a Position Sensitivity Index (PSI_p), reports controlled experiments on text-only (MS-MARCO), image-only (ChartQA), and mixed-modality (VEGA) settings with four vision-language models, and claims a consistent U-shaped accuracy curve, intensification of bias in multimodal settings, and a logarithmic growth of bias with retrieval size. The paper also presents attention visualization at a selected decoder layer to support a mechanism based on cross-modal attention sparsity. The authors recommend position-aware reweighting or debiasing strategies for multimodal RAG systems.
Significance. Position bias in RAG is practically important, and the paper addresses a genuine gap by extending position-bias analysis from text-only to multimodal settings. The controlled perturbation design and the PSI_p metric are useful analytical tools, and the comparison across text, image, and mixed modalities with several model families is valuable. If the central claims were fully supported, the paper would be a useful contribution to the reliability and interpretability of multimodal RAG. However, the current version contains internal numerical inconsistencies, and several headline claims are not supported by the paper's own data, so the contribution cannot be assessed as stated.
major comments (5)
- [Section 3.2, Table 3, Eq. (2)] The prose reporting the PSI_p growth with k does not match Table 3. The text states that when k increases from 3 to 20, MiniCPM-v2.6 rises from about 0.018 to 0.060, Qwen2-VL from 0.020 to 0.080, Llama-3.2 from 0.030 to 0.085, and GPT-4o from 0.028 to 0.082, whereas Table 3 reports k=3 values of 0.040, 0.032, 0.024, 0.107 and k=20 values of 0.119, 0.137, 0.118, 0.152. The reported least-squares fit for Llama-3.2 (alpha=0.035, beta=0.010, R2=0.986) cannot be reproduced from Table 3: at k=3 the predicted value is about 0.049 while the table gives 0.024, and at k=20 the prediction is about 0.125 while the table gives 0.118. Because Eq. (2) is the paper's central quantitative claim, the fit must be transparently reproducible from the published numbers, and the prose and table must be reconciled.
- [Section 2.2, Table 1] The claim of a 'consistent U-shaped accuracy curve' is contradicted by the paper's own Table 1 for Llama-3.2-11B-Vision, whose image-only accuracies decrease monotonically (0.2256, 0.2103, 0.2019) and whose mixed-modality accuracies also decrease monotonically (0.2496, 0.2272, 0.2202). Thus the universal U-shape stated in the abstract and conclusion is not supported by the full data. The authors should either qualify the claim to the models and settings where the U-shape holds or provide additional analysis (for example, per-dataset or per-model tests) explaining the exceptions.
- [Section 3.1, Table 2, Abstract] The claim that 'multimodal interactions intensify position bias compared to unimodal settings' is not supported by Table 2 for all models. For GPT-4o the text-only PSI is 0.0570 while the mixed-modality PSI is 0.0536, a decrease, and for Qwen2-VL the image-only PSI (0.0324) is lower than the text-only PSI (0.0372). The prose in Section 3.1 correctly says 'most models,' but the abstract and conclusion state the effect categorically. The authors should either restrict the claim to the models where it holds or provide a significance test and a clear statement of the exceptions.
- [Section 3.2, Eq. (2)] Equation (2) is fitted only on ChartQA (image-only) data for k in {3,5,10,15,20}, yet the paper generalizes the logarithmic law to text and mixed-modality RAG in the abstract and conclusion. No fit is reported for MS-MARCO or VEGA, and no test is shown that the text or mixed data follow the same log-linear trend. A five-point fit on a single dataset and modality is weak evidence for a general law. The authors should either fit the model on all three modalities and report the parameters and goodness-of-fit for each, or explicitly restate the claim as an image-only empirical observation.
- [Section 4.2, Section 4.3] The selection of layer 14 for attention visualization is circular with respect to the paper's mechanistic claims. The layer is chosen because it has the highest S2D sparsity, but the claim that middle layers are 'critical for cross-modal integration' is introduced as a hypothesis (Section 4.2 states 'we hypothesize'), and then the same layer is used in Section 4.3 to 'experimentally validate' position bias in cross-modal attention. The paper should provide an independent justification for why S2D measures cross-modal integration quality, or show that the positional bias pattern is robust across multiple layers rather than an artifact of the layer-selection criterion.
minor comments (6)
- [Section 2.2, Table 1] Table 1 heading says 'accuracy (%)' but all entries are proportions between 0 and 1; please make the units consistent.
- [Section 3.1, Eq. (1)] There is a typo in the sentence after Eq. (1): 'When p=1, p=1 and the highest/lowest positions...' should read 'When p=1, the highest/lowest positions...'.
- [Section 3.2, Table 3] The caption of Table 3 says 'mean ± variance' but the reported quantities are more naturally interpreted as standard deviations; please clarify what the ± values represent.
- [Section 3.1] The sentence 'LLAMa-3.2 shows a relatively higher sensitivity (PSI≈0.045) under the mixed-modality conditions' does not match Table 2, where the mixed-modality PSI for Llama-3.2 is 0.0294; the discrepancy should be corrected.
- [Section 4.1, Eq. (4)] The normalization in Eq. (4) divides by max(AROI)-min(AROI); if the ROI has constant attention, this is undefined. Please add a degenerate-case handling or state the assumption.
- [General] The paper would benefit from providing code, data splits, or a reproducibility statement, since the exact construction of the controlled triplets and the random perturbation protocol are not fully specified.
Circularity Check
No circular derivation: the PSI metric, U-shape, and log-law are empirical fits, not results that reduce to their own inputs by construction.
full rationale
The paper's central quantitative claims are empirical measurements. PSI_p (Eq. 1) is defined from measured accuracy at evidence positions and is not defined in terms of ln(k); Eq. 2 is an in-sample least-squares fit of PSI_p versus ln(k) on ChartQA. That is standard curve fitting, not a prediction forced by construction. The U-shaped accuracy curves are direct observations from controlled position permutations. The claim that multimodal interactions intensify bias is a comparison of measured PSI values across Table 2, not an identity. The layer-14 attention analysis is selected by an independent sparsity metric (N2E/S2D) before visualization, so the visualization is not used to define the layer choice. The paper contains no load-bearing self-citation: citations to Ge et al. and Bi et al. are contextual related-work, not premises of the derivation. Concerns raised in review—the mismatch between prose and Table 3 values, the five-point ChartQA-only fit, and generalization to text/mixed modalities—are threats to reproducibility and external validity, not circularity. No equation reduces to another by construction and no fitted parameter is renamed as a prediction. Score 0.
Assumptions & free parameters
free parameters (4)
- alpha (log-slope) and beta (intercept) for MiniCPM-v2.6 =
alpha between 0.030 and 0.040; beta not reported
- alpha (log-slope) and beta (intercept) for Qwen2-VL-7B =
alpha between 0.030 and 0.040; beta not reported
- alpha (log-slope) and beta (intercept) for Llama-3.2-11B =
alpha = 0.035, beta = 0.010
- alpha (log-slope) and beta (intercept) for GPT-4o =
alpha between 0.030 and 0.040; beta not reported
assumptions (5)
- domain assumption Answer accuracy on MS-MARCO, ChartQA, and VEGA can be scored with a single comparable 'accuracy' metric, though the paper never defines it.
- domain assumption Manually constructed triplets (one gold, two distractors) faithfully simulate real RAG retrieval ordering.
- domain assumption Ten randomized repetitions per configuration are sufficient to separate systematic position bias from sampling noise.
- ad hoc to paper Least-squares fit of PSIp vs ln(k) on ChartQA data evidences a general scaling law across modalities.
- ad hoc to paper Layer 14 is the optimal layer for cross-modal attention analysis because it exhibits the highest sparsity (S2D).
Cite this review
Pith. "Pith review of Who is in the Spotlight: The Hidden Bias Undermining Multimodal Retrieval-Augmented Generation." pith.science (2026). https://pith.science/paper/YJT2PS3G
@misc{pith2026250611063,
author = {Pith},
title = {Pith review of: Who is in the Spotlight: The Hidden Bias Undermining Multimodal Retrieval-Augmented Generation},
year = {2026},
howpublished = {\url{https://pith.science/paper/YJT2PS3G}},
note = {Machine review of arXiv:2506.11063}
}
abstract
Multimodal Retrieval-Augmented Generation (RAG) systems have become essential in knowledge-intensive and open-domain tasks. As retrieval complexity increases, ensuring the robustness of these systems is critical. However, current RAG models are highly sensitive to the order in which evidence is presented, often resulting in unstable performance and biased reasoning, particularly as the number of retrieved items or modality diversity grows. This raises a central question: How does the position of retrieved evidence affect multimodal RAG performance? To answer this, we present the first comprehensive study of position bias in multimodal RAG systems. Through controlled experiments across text-only, image-only, and mixed-modality tasks, we observe a consistent U-shaped accuracy curve with respect to evidence position. To quantify this bias, we introduce the Position Sensitivity Index ($PSI_p$) and develop a visualization framework to trace attention allocation patterns across decoder layers. Our results reveal that multimodal interactions intensify position bias compared to unimodal settings, and that this bias increases logarithmically with retrieval range. These findings offer both theoretical and empirical foundations for position-aware analysis in RAG, highlighting the need for evidence reordering or debiasing strategies to build more reliable and equitable generation systems.
Figures
Figures from the paper (2 more)
Reference graph
Works this paper leans on
-
[1]
Mohammad Mahdi Abootorabi, Amirhosein Zobeiri, Mahdi Dehghani, Mohammadali Mohammadkhani, Bardia Mohammadi, Omid Ghahroodi, Mahdieh Soleymani Baghshah, and Ehsaneddin Asgari. 2025. https://arxiv.org/abs/2502.08826 Ask in any modality: A comprehensive survey on multimodal retrieval-augmented generation . Preprint, arXiv:2502.08826
arXiv 2025
-
[2]
AI@Meta. 2024. https://github.com/meta-llama/llama3/blob/main/MODEL_CARD.md Llama 3 model card
2024
-
[3]
Baolong Bi, Shaohan Huang, Yiwei Wang, Tianchi Yang, Zihan Zhang, Haizhen Huang, Lingrui Mei, Junfeng Fang, Zehao Li, Furu Wei, and 1 others. 2024. Context-dpo: Aligning language models for context-faithfulness. arXiv preprint arXiv:2412.15280
arXiv 2024
-
[4]
Jo \ a o Coelho, Bruno Martins, Joao Magalhaes, Jamie Callan, and Chenyan Xiong. 2024. https://doi.org/10.18653/v1/2024.acl-short.35 Dwell in the beginning: How language models embed long documents for dense retrieval . In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 370--377, Bang...
-
[5]
Darren Edge, Ha Trinh, Newman Cheng, Joshua Bradley, Alex Chao, Apurva Mody, Steven Truitt, Dasha Metropolitansky, Robert Osazuwa Ness, and Jonathan Larson. 2025. https://arxiv.org/abs/2404.16130 From local to global: A graph rag approach to query-focused summarization . Preprint, arXiv:2404.16130
arXiv 2025
-
[6]
Manuel Faysse, Hugues Sibille, Tony Wu, Bilel Omrani, Gautier Viaud, Céline Hudelot, and Pierre Colombo. 2024. https://arxiv.org/abs/2407.01449 Colpali: Efficient document retrieval with vision language models . Preprint, arXiv:2407.01449
arXiv 2024
-
[7]
Yuyao Ge, Shenghua Liu, Baolong Bi, Yiwei Wang, Lingrui Mei, Wenjie Feng, Lizhe Chen, and Xueqi Cheng. 2024. https://arxiv.org/abs/2402.07140 Can graph descriptive order affect solving graph problems with llms? Preprint, arXiv:2402.07140
arXiv 2024
-
[8]
Kun Luo, Zheng Liu, Shitao Xiao, Tong Zhou, Yubo Chen, Jun Zhao, and Kang Liu. 2024. https://doi.org/10.18653/v1/2024.acl-long.180 Landmark embedding: A chunking-free embedding method for retrieval augmented long-context large language models . In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Paper...
Show all 29 references
-
[9]
Ahmed Masry, Do Xuan Long, Jia Qing Tan, Shafiq Joty, and Enamul Hoque. 2022. https://doi.org/10.18653/v1/2022.findings-acl.177 C hart QA : A benchmark for question answering about charts with visual and logical reasoning . In Findings of the Association for Computational Ling...
2022 doi
-
[10]
Tri Nguyen, Mir Rosenberg, Xia Song, Jianfeng Gao, Saurabh Tiwary, Rangan Majumder, and Li Deng. 2016. https://arxiv.org/abs/1611.09268 MS MARCO: A human generated machine reading comprehension dataset . CoRR, abs/1611.09268
2016 arXiv
-
[11]
OpenAI. 2024. https://openai.com/index/hello-gpt-4o/ Hello, gpt-4o | openai
2024
-
[12]
OpenBMB. 2024. https://huggingface.co/openbmb/MiniCPM-V-2\_6 openbmb/minicpm-v-2\_6
2024
-
[13]
Aleksander Theo Strand, Sushant Gautam, Cise Midoglu, and Pål Halvorsen. 2024. https://arxiv.org/abs/2406.01273 Soccerrag: Multimodal soccer information retrieval via natural queries . Preprint, arXiv:2406.01273
2024 arXiv
-
[14]
Zhijie Tan, Xu Chu, Weiping Li, and Tong Mo. 2024. https://arxiv.org/abs/2410.16983 Order matters: Exploring order sensitivity in multimodal large language models . Preprint, arXiv:2410.16983
2024 arXiv
-
[15]
Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Yang Fan, Kai Dang, Mengfei Du, Xuancheng Ren, Rui Men, Dayiheng Liu, Chang Zhou, Jingren Zhou, and Junyang Lin. 2024 a . Qwen2-vl: Enhancing vision-language m...
2024 arXiv
-
[16]
Yiwei Wang, Yujun Cai, Muhao Chen, Yuxuan Liang, and Bryan Hooi. 2024 b . https://arxiv.org/abs/2310.13206 Primacy effect of chatgpt . Preprint, arXiv:2310.13206
2024 arXiv
-
[17]
Siwei Wu, Kang Zhu, Yu Bai, Yiming Liang, Yizhi Li, Haoning Wu, Jiaheng Liu, Ruibo Liu, Xingwei Qu, Xuxin Cheng, and 1 others. 2024. Mmra: A benchmark for multi-granularity multi-image relational association. arXiv preprint arXiv:2407.17379
2024 arXiv
-
[18]
Peng Xia, Kangyu Zhu, Haoran Li, Tianze Wang, Weijia Shi, Sheng Wang, Linjun Zhang, James Zou, and Huaxiu Yao. 2024 a . Mmed-rag: Versatile multimodal rag system for medical vision language models. arXiv preprint arXiv:2410.13085
2024 arXiv
-
[19]
Peng Xia, Kangyu Zhu, Haoran Li, Hongtu Zhu, Yun Li, Gang Li, Linjun Zhang, and Huaxiu Yao. 2024 b . https://doi.org/10.18653/v1/2024.emnlp-main.62 RULE : Reliable multimodal RAG for factuality in medical vision language models . In Proceedings of the 2024 Conference on Empiri...
2024 doi
-
[20]
Soyoung Yoon, Eunbi Choi, Jiyeon Kim, Hyeongu Yun, Yireun Kim, and Seung-won Hwang. 2024. https://doi.org/10.18653/v1/2024.acl-long.125 L ist T 5: Listwise reranking with fusion-in-decoder improves zero-shot retrieval . In Proceedings of the 62nd Annual Meeting of the Associat...
2024 doi
-
[21]
Shi Yu, Chaoyue Tang, Bokai Xu, Junbo Cui, Junhao Ran, Yukun Yan, Zhenghao Liu, Shuo Wang, Xu Han, Zhiyuan Liu, and Maosong Sun. 2025. https://arxiv.org/abs/2410.10594 Visrag: Vision-based retrieval-augmented generation on multi-modality documents . Preprint, arXiv:2410.10594
2025 arXiv
-
[22]
Jianhao Yuan, Shuyang Sun, Daniel Omeiza, Bo Zhao, Paul Newman, Lars Kunze, and Matthew Gadd. 2024. Rag-driver: Generalisable driving explanations with retrieval-augmented in-context learning in multi-modal large language model. arXiv preprint arXiv:2402.10828
2024
-
[23]
Meiru Zhang, Zaiqiao Meng, and Nigel Collier. 2024. https://doi.org/10.18653/v1/2024.findings-emnlp.732 Can we instruct LLM s to compensate for position bias? In Findings of the Association for Computational Linguistics: EMNLP 2024, pages 12545--12556, Miami, Florida, USA. Ass...
2024 doi
-
[24]
Xiangyu Zhao, Yuehan Zhang, Wenlong Zhang, and Xiao-Ming Wu. 2024. https://api.semanticscholar.org/CorpusID:271915652 Unifashion: A unified vision-language model for multimodal fashion retrieval and generation . In Conference on Empirical Methods in Natural Language Processing
2024
-
[25]
Chenyu Zhou, Mengdan Zhang, Peixian Chen, Chaoyou Fu, Yunhang Shen, Xiawu Zheng, Xing Sun, and Rongrong Ji. 2024 a . https://arxiv.org/abs/2406.10228 Vega: Learning interleaved image-text comprehension in vision-language large models . Preprint, arXiv:2406.10228
2024 arXiv
-
[26]
Xu, Zhiruo Wang, Zhengbao Jiang, and Graham Neubig
Shuyan Zhou, Uri Alon, Frank F. Xu, Zhiruo Wang, Zhengbao Jiang, and Graham Neubig. 2023. https://arxiv.org/abs/2207.05987 Docprompting: Generating code by retrieving the docs . In International Conference on Learning Representations (ICLR), Kigali, Rwanda
2023 arXiv
-
[27]
Zhongliang Zhou, Jielu Zhang, Zihan Guan, Mengxuan Hu, Ni Lao, Lan Mu, Sheng Li, and Gengchen Mai. 2024 b . Img2loc: Revisiting image geolocalization using multi-modality foundation models and image-based retrieval-augmented generation. In Proceedings of the 47th International...
2024
-
[28]
online" 'onlinestring :=
ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list...
-
[29]
write newline
" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...
Reviewed August 7, 2026 · model on record in the stance chip above.
Discussion (0). Sign in to comment.