Subtraction Gets You More: Gap-Aware Retrieval for Multimodal Multi-Hop QA

Jay-Yoon Lee; Sunah O

arxiv: 2605.28641 · v2 · pith:GDHPY2XMnew · submitted 2026-05-27 · 💻 cs.IR

Subtraction Gets You More: Gap-Aware Retrieval for Multimodal Multi-Hop QA

Sunah O , Jay-Yoon Lee This is my paper

Pith reviewed 2026-06-29 09:40 UTC · model grok-4.3

classification 💻 cs.IR

keywords multimodal QAmulti-hop retrievaliterative retrievalquery embeddingsemantic anchoringgap-aware retrievalevidence completion

0 comments

The pith

Subtracting the embedding of retrieved context from the query embedding allows retrievers to locate missing evidence instead of repeating similar items in multimodal multi-hop question answering.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper establishes that conventional iterative retrieval in multimodal multi-hop QA suffers from semantic anchoring, where fetched evidence causes the system to retrieve redundant, entity-centric items. To address this, it introduces a method of implicit query rewriting at the embedding level through context subtraction. This gap-aware approach is shown to support better compositional reasoning across modalities. When routed dynamically with additive updates for other tasks, the resulting hybrid system achieves substantial gains on the MultimodalQA benchmark. The result suggests that the direction of embedding updates matters for expanding the effective search space without external supervision.

Core claim

By performing context-subtractive query steering directly in embedding space, GRAIL breaks the semantic anchoring trap that causes iterative retrievers to produce entity-centric redundancy. Subtractive updates prove effective for evidence set completion tasks requiring compositional cross-modal reasoning, while additive updates suit sequential pool construction for localized aggregation. The hybrid framework that selects the update type based on task achieves a 40.3% macro-averaged performance gain on MultimodalQA, with sequential GRAIL demonstrating superior noise resilience and an expanded search horizon.

What carries the argument

GRAIL (Gap-aware Retrieval via Adaptive Implicit Localization), the paradigm of implicit query rewriting at the embedding level using context-subtractive steering.

If this is right

Context-subtractive steering excels at compositional cross-modal reasoning.
Additive embedding updates show strength on localized information aggregation.
The hybrid framework achieves a 40.3% macro-averaged performance gain on MultimodalQA.
Sequential GRAIL retrieves in a superior, noise-resilient manner and expands the search horizon.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

The subtraction technique could be applied to single-modal text retrieval to test generality.
It may interact with different embedding models in ways that affect performance.
Further iterations beyond the tested ones might yield even larger gains in complex queries.

Load-bearing premise

That performing subtraction directly on embeddings will break the semantic anchoring trap without introducing new retrieval failure modes.

What would settle it

Running the method on a dataset where semantic anchoring is measured by entity overlap and checking if overlap decreases compared to baseline iterative retrieval.

Figures

Figures reproduced from arXiv: 2605.28641 by Jay-Yoon Lee, Sunah O.

**Figure 1.** Figure 1: Redundancy trap in MMQA. Given a table and three emblems, the goal is to fetch the missing visual evidence (Bolton Wanderers’s emblem) to complete the evidence set. (i) Query + Context fail as they trapped by the existing Lancashire Senior Cup context (Semantic Anchoring). Meanwhile, Explicit Rewriting (IRCoT) prematurely concludes the search upon verifying Manchester City’s record, overlooking the unexami… view at source ↗

**Figure 2.** Figure 2: Hybrid framework leveraging complementary specialists in MultimodalQA. By dynamically routing each query to the optimal specialist, our hybrid framework achieves a "best-of-both-worlds" performance (+40.3% macro avg. gain over Additive) [PITH_FULL_IMAGE:figures/full_fig_p007_2.png] view at source ↗

**Figure 3.** Figure 3: Robustness of GRAIL vs. Additive models under iterative slicing. Set-Recall(%) is plotted against the number of retrieval slices in MultimodalQA for K = 10 (left) and K = 20 (right). Although the Additive baseline exhibits strength in specific single-stage question types ( [PITH_FULL_IMAGE:figures/full_fig_p008_3.png] view at source ↗

**Figure 4.** Figure 4: Additional Case Study. Given an image of Lons-le-Saunier, the goal is to fetch the missing table evidence (the list of principal cities of Franche-Comte) to complete the evidence set. from the target Franche-Comté table. However, this case also uncovers a nuanced over-subtraction phenomenon: while the operation successfully surfaces the relevant table structure, erasing the visual features of Lons-le-Saun… view at source ↗

**Figure 5.** Figure 5: Taxonomy of cross-modal semantic alignment. Top row shows query-independent approaches: (a) Centroid-based alignment, (b) CLS-based alignment. Bottom row shows query-dependent approaches: (c) Query-evidence alignment, (d) Query-set alignment [PITH_FULL_IMAGE:figures/full_fig_p012_5.png] view at source ↗

**Figure 6.** Figure 6: Model specialization by task complexity in MultimodalQA. Additive models excel in Uni-modal lookup, while Gap-aware models dominate reasoningheavy tasks (Compare, Compose). This performance gap remains consistent across alignment architectures. 15.68% to 8.30% (with ∆esc turning to −0.034), full parameter updates break the highly precise, cross-modal arrangements Therefore, as established in our main exp… view at source ↗

**Figure 8.** Figure 8: illustrates the incremental Set-Recall@10 gains achieved exclusively during the sequential rescue steps (partitioning scheme 3+2+3+2). The results show that our GRAIL consistently delivers the highest retrieval gains across all task types except Uni-modal. Particularly in the Compare category, GRAIL outperforms the Query Only baseline by 4.5pp and the Additive model by 11.2pp. This stark gap stems from t… view at source ↗

**Figure 9.** Figure 9: Hybrid framework leveraging complementary specialists in WebQA. By dynamically routing each query to the optimal specialist, our Hybrid framework achieves a "best-of-both-worlds" performance. (+195.7% micro avg. gain over Additive) [PITH_FULL_IMAGE:figures/full_fig_p017_9.png] view at source ↗

read the original abstract

In multimodal multi-hop question answering, we focus on the initial retrieval stage via two distinct tasks: (1) evidence set completion, retrieving missing evidence given context, and (2) sequential pool construction, iteratively building the top-$K$ pool from the scratch. Under these settings, we point out that conventional iterative retrieval frameworks often suffer from Semantic Anchoring, where previously fetched evidence traps the retriever and yields entity-centric redundancy. To break this trap, we propose GRAIL (Gap-aware Retrieval via Adaptive Implicit Localization), a paradigm that performs implicit query rewriting directly at the embedding level. By context-subtractive query steering, GRAIL excels at compositional cross-modal reasoning, while additive embedding updates show strength on localized information aggregation. By dynamically routing queries based on task type, our Hybrid Framework achieves a 40.3% macro-averaged performance gain on MultimodalQA. Extensive evaluations demonstrate that sequential GRAIL retrieves in a superior, noise-resilient manner, significantly expanding the search horizon through iterative gap-aware optimization.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Desk Editor's Note private letter to a colleague

GRAIL's embedding subtraction for gap-aware retrieval in multimodal QA is a coherent incremental idea but the 40% gain claim needs full experimental details to evaluate.

read the letter

The main takeaway is that this paper proposes GRAIL, which applies subtraction to query embeddings to make retrieval more aware of gaps in multimodal multi-hop QA. It claims this helps avoid semantic anchoring where prior evidence causes redundant fetches.

What is new is the adaptive implicit localization via context-subtractive steering, contrasted with additive updates, and the hybrid routing for different tasks. The description of the problem and the two settings (evidence completion and sequential pool construction) is clear and points to a genuine limitation in standard iterative approaches.

The paper does well in highlighting how subtraction expands the search horizon in a noise-resilient way. If the full results support the 40.3% macro gain with proper controls, that would be a solid incremental advance.

The soft spots are around the lack of visible experimental rigor in the summary: no baselines listed, no error analysis, and no check on whether the routing introduces its own biases. The central claim rests on empirical results that need to be inspected closely for fairness.

This paper is for retrieval researchers working on vision-language QA systems. Someone building or evaluating multi-hop retrievers would get value from the method if it holds up.

I recommend sending it for peer review. The idea is grounded enough in the problem description to merit referee attention, though the evaluation details will determine how much impact it has.

Referee Report

2 major / 1 minor

Summary. The paper proposes GRAIL (Gap-aware Retrieval via Adaptive Implicit Localization), which performs implicit query rewriting at the embedding level via context-subtractive steering to mitigate Semantic Anchoring in conventional iterative retrieval frameworks for multimodal multi-hop QA. It distinguishes evidence set completion and sequential pool construction tasks, introduces additive vs. subtractive embedding updates, and presents a Hybrid Framework that routes queries by task type, claiming a 40.3% macro-averaged performance gain on MultimodalQA with noise-resilient expansion of the search horizon.

Significance. If the empirical claims hold after proper validation, the embedding-level subtraction mechanism could provide a lightweight way to reduce entity-centric redundancy in multi-hop retrieval without explicit rewriting models. The task-aware hybrid routing is a plausible design choice, but the absence of any supporting experiments, ablations, or formal definitions in the manuscript prevents assessing whether the approach actually expands search horizons or introduces new failure modes.

major comments (2)

[Abstract] Abstract: The central claim of a 40.3% macro-averaged performance gain on MultimodalQA is stated with no experimental details, baselines, dataset splits, error bars, ablation studies, or results tables, rendering the claim impossible to evaluate or reproduce.
[Abstract] Abstract: No equations, pseudocode, or derivation is given for context-subtractive query steering, gap-aware optimization, or the implicit localization mechanism, so the technical contribution cannot be assessed for correctness or novelty relative to existing embedding arithmetic methods.

minor comments (1)

[Abstract] Abstract: The phrase 'Semantic Anchoring' is introduced without a formal definition or citation to related concepts such as query drift or embedding collapse.

Simulated Author's Rebuttal

2 responses · 0 unresolved

We appreciate the referee's feedback on our submission. We respond to the major comments below and will make revisions to the manuscript to address the identified issues.

read point-by-point responses

Referee: [Abstract] Abstract: The central claim of a 40.3% macro-averaged performance gain on MultimodalQA is stated with no experimental details, baselines, dataset splits, error bars, ablation studies, or results tables, rendering the claim impossible to evaluate or reproduce.

Authors: We agree that the abstract is too concise and omits key experimental details needed for evaluation. In the revised manuscript we will expand the abstract to briefly note the MultimodalQA dataset, the hybrid routing setup, the baselines compared, and explicit references to the full results tables, error bars, and ablation studies presented in Sections 4 and 5. revision: yes
Referee: [Abstract] Abstract: No equations, pseudocode, or derivation is given for context-subtractive query steering, gap-aware optimization, or the implicit localization mechanism, so the technical contribution cannot be assessed for correctness or novelty relative to existing embedding arithmetic methods.

Authors: We acknowledge that the abstract provides only a high-level description. We will add the formal equations for context-subtractive embedding updates (query' = query - context), the gap-aware optimization objective, and pseudocode for the hybrid task-aware routing and iterative retrieval procedure to the Methods section so that correctness and novelty relative to prior embedding arithmetic work can be directly assessed. revision: yes

Circularity Check

0 steps flagged

No significant circularity; derivation chain absent

full rationale

The provided text consists solely of an abstract describing an empirical method (GRAIL with context-subtractive steering and hybrid routing) and reported performance gains (40.3% on MultimodalQA). No equations, parameters, derivations, or self-citations appear. None of the enumerated circularity patterns apply because there is no derivation chain to reduce to inputs by construction. The central claims are presented as experimental outcomes rather than mathematical predictions or fitted renamings. This is the expected honest non-finding for a methods paper whose full text (per the query) contains no load-bearing self-referential steps.

Axiom & Free-Parameter Ledger

0 free parameters · 0 axioms · 0 invented entities

Abstract-only review; no free parameters, axioms, or invented entities are specified in the provided text.

pith-pipeline@v0.9.1-grok · 5702 in / 1203 out tokens · 23768 ms · 2026-06-29T09:40:38.178783+00:00 · methodology

discussion (0)

Reference graph

Works this paper leans on

12 extracted references · 4 canonical work pages · 2 internal anchors

[1]

InFindings of the Association for Computational Linguistics: NAACL 2025, pages 1061–1076

Ramqa: A unified framework for retrieval-augmented multi-modal question answering. InFindings of the Association for Computational Linguistics: NAACL 2025, pages 1061–1076. Wenhu Chen, Hexiang Hu, Xi Chen, Pat Verga, and William Cohen

2025
[2]

InProceedings of the 2022 Conference on Empirical Methods in Natural Lan- guage Processing, pages 5558–5570

Murag: Multimodal retrieval- augmented generator for open question answering over images and text. InProceedings of the 2022 Conference on Empirical Methods in Natural Lan- guage Processing, pages 5558–5570. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova

2022
[3]

Bert: Pre-training of deep bidirectional transformers for language understand- ing. InProceedings of the 2019 conference of the North American chapter of the association for com- putational linguistics: human language technologies, volume 1 (long and short papers), pages 4171–4186. OpenAI Aaron Hurst, Adam Lerer, Adam P. Goucher, Adam Perelman, Aditya Ram...

2019
[4]

Unsupervised Dense Information Retrieval with Contrastive Learning

Unsupervised dense in- formation retrieval with contrastive learning.arXiv preprint arXiv:2112.09118. Vladimir Karpukhin, Barlas Oguz, Sewon Min, Patrick Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih

work page internal anchor Pith review Pith/arXiv arXiv
[5]

InProceedings of the 2020 conference on empirical methods in natural language processing (EMNLP), pages 6769–6781

Dense passage retrieval for open- domain question answering. InProceedings of the 2020 conference on empirical methods in natural language processing (EMNLP), pages 6769–6781. Peng Qi, Xiaowen Lin, Leo Mehr, Zijian Wang, and Christopher D Manning

2020
[6]

Answering complex open-domain questions through iterative query gen- eration. InProceedings of the 2019 conference on empirical methods in natural language processing and the 9th international joint conference on natu- ral language processing (EMNLP-IJCNLP), pages 2590–2602. Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini ...

2019
[7]

C-Pack: Packed Resources For General Chinese Embeddings

C-pack: Packaged resources to advance general chinese embedding.Preprint, arXiv:2309.07597. Lee Xiong, Chenyan Xiong, Ye Li, Kwok-Fung Tang, Jialin Liu, Paul Bennett, Junaid Ahmed, and Arnold Overwijk

work page internal anchor Pith review Pith/arXiv arXiv
[8]

Approximate Nearest Neighbor Negative Contrastive Learning for Dense Text Retrieval

Approximate nearest neighbor neg- ative contrastive learning for dense text retrieval. arXiv preprint arXiv:2007.00808. Wenhan Xiong, Xiang Lorraine Li, Srinivasan Iyer, Jingfei Du, Patrick Lewis, William Yang Wang, Yashar Mehdad, Wen-tau Yih, Sebastian Riedel, Douwe Kiela, and Barlas O ˘guz

work page Pith review arXiv 2007
[9]

InInternational Conference on Learning Representations, volume 2025, pages 72914–72938

Infer- ence scaling for long-context retrieval augmented generation. InInternational Conference on Learning Representations, volume 2025, pages 72914–72938. Joohyung Yun, Doyup Lee, and Wook-Shin Han

2025
[10]

Pon- tarlier

Lilac: Late interacting in layered component graph for open-domain multimodal multihop retrieval. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pages 20551–20570. A Additional Qualitative Case Study To further illustrate the practical mechanics of GRAIL, we provide a detailed visualization of a representative m...

2025
[11]

Images are embedded using CLIP ViT-B/32 (Radford et al., 2021)

to embed questions, text passages, and table rows. Images are embedded using CLIP ViT-B/32 (Radford et al., 2021). All embeddings are projected into a shared latent space of dimension d= 1024 . We train the retrieval model using the AdamW optimizer with a batch size of 32, 100 epochs, and a learning rate of1e−4 . The base temperature τbase is set to 0.05....

2021
[12]

##; … ; #$ Transformer [QA] TOKEN B Image Text contrastive penalty from negative anchor pull to anchor TOKEN A TOKEN C [CLS] [CLS] [CLS] [CLS]

fine-tuned independently for 3 epochs with a learn- ing rate of1e−5. C Detailed Alignment Strategies Analysis This section extends the empirical evaluation of Section 4.1 by providing a comprehensive analy- sis of the interaction between cross-modal align- ment topologies, training regimes (Frz vs. FT), and downstream task complexities. C.1 Cross-Examinat...

work page arXiv

[1] [1]

InFindings of the Association for Computational Linguistics: NAACL 2025, pages 1061–1076

Ramqa: A unified framework for retrieval-augmented multi-modal question answering. InFindings of the Association for Computational Linguistics: NAACL 2025, pages 1061–1076. Wenhu Chen, Hexiang Hu, Xi Chen, Pat Verga, and William Cohen

2025

[2] [2]

InProceedings of the 2022 Conference on Empirical Methods in Natural Lan- guage Processing, pages 5558–5570

Murag: Multimodal retrieval- augmented generator for open question answering over images and text. InProceedings of the 2022 Conference on Empirical Methods in Natural Lan- guage Processing, pages 5558–5570. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova

2022

[3] [3]

Bert: Pre-training of deep bidirectional transformers for language understand- ing. InProceedings of the 2019 conference of the North American chapter of the association for com- putational linguistics: human language technologies, volume 1 (long and short papers), pages 4171–4186. OpenAI Aaron Hurst, Adam Lerer, Adam P. Goucher, Adam Perelman, Aditya Ram...

2019

[4] [4]

Unsupervised Dense Information Retrieval with Contrastive Learning

Unsupervised dense in- formation retrieval with contrastive learning.arXiv preprint arXiv:2112.09118. Vladimir Karpukhin, Barlas Oguz, Sewon Min, Patrick Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih

work page internal anchor Pith review Pith/arXiv arXiv

[5] [5]

InProceedings of the 2020 conference on empirical methods in natural language processing (EMNLP), pages 6769–6781

Dense passage retrieval for open- domain question answering. InProceedings of the 2020 conference on empirical methods in natural language processing (EMNLP), pages 6769–6781. Peng Qi, Xiaowen Lin, Leo Mehr, Zijian Wang, and Christopher D Manning

2020

[6] [6]

Answering complex open-domain questions through iterative query gen- eration. InProceedings of the 2019 conference on empirical methods in natural language processing and the 9th international joint conference on natu- ral language processing (EMNLP-IJCNLP), pages 2590–2602. Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini ...

2019

[7] [7]

C-Pack: Packed Resources For General Chinese Embeddings

C-pack: Packaged resources to advance general chinese embedding.Preprint, arXiv:2309.07597. Lee Xiong, Chenyan Xiong, Ye Li, Kwok-Fung Tang, Jialin Liu, Paul Bennett, Junaid Ahmed, and Arnold Overwijk

work page internal anchor Pith review Pith/arXiv arXiv

[8] [8]

Approximate Nearest Neighbor Negative Contrastive Learning for Dense Text Retrieval

Approximate nearest neighbor neg- ative contrastive learning for dense text retrieval. arXiv preprint arXiv:2007.00808. Wenhan Xiong, Xiang Lorraine Li, Srinivasan Iyer, Jingfei Du, Patrick Lewis, William Yang Wang, Yashar Mehdad, Wen-tau Yih, Sebastian Riedel, Douwe Kiela, and Barlas O ˘guz

work page Pith review arXiv 2007

[9] [9]

InInternational Conference on Learning Representations, volume 2025, pages 72914–72938

Infer- ence scaling for long-context retrieval augmented generation. InInternational Conference on Learning Representations, volume 2025, pages 72914–72938. Joohyung Yun, Doyup Lee, and Wook-Shin Han

2025

[10] [10]

Pon- tarlier

Lilac: Late interacting in layered component graph for open-domain multimodal multihop retrieval. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pages 20551–20570. A Additional Qualitative Case Study To further illustrate the practical mechanics of GRAIL, we provide a detailed visualization of a representative m...

2025

[11] [11]

Images are embedded using CLIP ViT-B/32 (Radford et al., 2021)

to embed questions, text passages, and table rows. Images are embedded using CLIP ViT-B/32 (Radford et al., 2021). All embeddings are projected into a shared latent space of dimension d= 1024 . We train the retrieval model using the AdamW optimizer with a batch size of 32, 100 epochs, and a learning rate of1e−4 . The base temperature τbase is set to 0.05....

2021

[12] [12]

##; … ; #$ Transformer [QA] TOKEN B Image Text contrastive penalty from negative anchor pull to anchor TOKEN A TOKEN C [CLS] [CLS] [CLS] [CLS]

fine-tuned independently for 3 epochs with a learn- ing rate of1e−5. C Detailed Alignment Strategies Analysis This section extends the empirical evaluation of Section 4.1 by providing a comprehensive analy- sis of the interaction between cross-modal align- ment topologies, training regimes (Frz vs. FT), and downstream task complexities. C.1 Cross-Examinat...

work page arXiv