UniHEAR: Unified Heterogeneous-Source Attentive Retrieval for Knowledge-Based Visual Question Answering

Ganzhong Luo; Hanyong Wang; Menglong Yang; Shuyu Zheng; Yang Ren

REVIEW 2 major objections 5 minor 40 references

UniHEAR: Unified Heterogeneous-Source Attentive Retrieval for Knowledge-Based Visual Question Answering

T0 review · 2 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Fusing image-to-image and image-to-text retrieval into one candidate pool, with reranker attention conditioned on coarse retrieval statistics, beats single-source retrieval and heavier BLIP-2 rerankers in knowledge-based VQA.

desk verdict Solid KB-VQA reranking paper that fuses I2I/I2T sources well and ablates thoroughly; the main fix is adding a simple heterogeneous-source fusion baseline and error bars. read the letter →

arxiv 2608.01147 v2 pith:VGQD7TC3 submitted 2026-08-02 cs.IR cs.CLcs.CV

Ganzhong Luo , Yang Ren , Hanyong Wang , Shuyu Zheng , Menglong Yang This is my paper

classification cs.IRcs.CLcs.CV

keywords knowledge-basedvisualquestionansweringheterogeneous-sourceretrievalmultimodalrerankingretrieval-augmentedgenerationcoarsedescriptormodalitygatingentitysourcefusion

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

UniHEAR is a retrieval-augmented framework for knowledge-based visual question answering that deliberately pulls candidate entities from two complementary sources at once: matching the query image against entity images (image-to-image) and against textual summaries (image-to-text), then reranking the union of both candidate lists. The paper's central claim is that telling the reranker which source each candidate came from, and how reliable that source's retrieval distribution was, fixes two systemic failures of prior systems: single-source retrieval that misses answers only the other modality can surface, and rerankers that waste capacity re-checking modalities coarse retrieval has already exploited. On E-VQA and InfoSeek, UniHEAR reports Recall@1 gains of 6.7 and 1.2 points over the strongest prior rerankers, and its reranker has 197M parameters, roughly six to nine times fewer than the BLIP-2 based competitors it outperforms. The bet that gives the approach its leverage is that cheap retrieval statistics, rank, similarity, and source-level entropy, can guide a lightweight reranker as effectively as a much larger model's own reasoning.

What carries the argument

The load-bearing object is the Coarse Retrieval Descriptor $D(e)\in\mathbb{R}^{10}$, a per-candidate vector that packages what the coarse retrievers already learned: normalized rank $r_s(e)$, raw similarity $\operatorname{sim}^s_c(e)$, z-normalized similarity $z_s(e)$, gap to the top-ranked entity $\delta_s(e)$, and pool-level entropy $H_s=-\sum_i p_i\log p_i$ over the top-$k$ retrieval distribution, for each source $s\in\{v,t\}$. It carries the argument because it turns a source-blind reranker into a source-aware one: through affine modulation $\gamma(D),\beta(D)$ it rescales and shifts the modality attention logits (Retrieval-Guided Attentive Modality Gating), and through the weight $w_s(e)=r_s(e)/(H_s+\varepsilon)$ it decides which source's coarse scores to trust (Entropy-Weighted Source Fusion), with the text-scale mismatch corrected by the pool-mean ratio $R=\mu_v/\mu_t$. The paper's ablations show that removing any of the five descriptor components degrades Recall@1, with the z-normalized similarity and the pool entropy being the two most informative.

What would settle it

Run the full pipeline on both benchmarks with Entropy-Weighted Source Fusion disabled, keeping the candidate union, the Coarse Retrieval Descriptor, and the gating intact, and compare Recall@1. If InfoSeek's drop is far smaller than E-VQA's, the entropy- and rank-weighted fusion is overfit to E-VQA's 2M-article distribution rather than a general source-reliability measure; if the drop is comparable, the claimed fusion mechanism transfers. A second observation: track whether ground-truth entities that rank well in the high-entropy (discounted) source are systematically suppressed, the failure mode the paper's own case study documents.

Watch

Extended reading notes

Core claim

The claim is that heterogeneous-source retrieval plus retrieval-guided reranking, both built around a per-candidate Coarse Retrieval Descriptor, removes the Single-Source Retrieval Bottleneck and the Retrieval-Source-Blind Reranking problem in KB-VQA. Each candidate entity is summarized by a ten-dimensional descriptor collecting its normalized rank, coarse similarity, z-normalized similarity, gap to the top-ranked entity, and pool-level entropy for both the image and the text retrieval source. A Retrieval-Guided Attentive Modality Gating module conditions the attention weights over image-text, text-only, and image-only representations on this descriptor, so a candidate already confirmed by visual retrieval leans on textual evidence instead of re-using the visual modality, while Entropy-Weighted Source Fusion adds the calibrated coarse scores back into the final ranking. A hybrid contrastive-plus-auxiliary training objective lets the same 197M-parameter model also do section-level retrieval, removing the need for a separate section reranker. The paper reports state-of-the-art Recall@1 of 49.5 on E-VQA (+6.7 over the best single-source reranker) and 65.2 on InfoSeek (+1.2).

Load-bearing premise

The paper assumes that the entropy of a retrieval source's top-$k$ score distribution, combined with per-candidate rank weighting and a single pool-mean calibration ratio, is a reliable and transferable measure of how much to trust that source, a premise validated by ablations on E-VQA but not on InfoSeek.

Editorial extensions

If this is right

The ceiling of retrieval-based KB-VQA rises: on E-VQA at $k=20$, the heterogeneous-source oracle reaches 70.1% Recall@5 versus 58.7% for image-to-text and 48.8% for image-to-image alone, and UniHEAR approaches that oracle.
A 197M-parameter reranker whose gating adds only 0.59M parameters can beat BLIP-2 based rerankers of 1.2-1.7B parameters, so raw reranker scale is not what decides retrieval quality once coarse priors are injected.
Entity-level and section-level retrieval can share one model: the auxiliary modality-preserving loss eliminates the standalone section reranker used by OMGM while improving section-level Recall@1.
Retrieval quality alone can move VQA accuracy without generator fine-tuning: UniHEAR with frozen Qwen2.5-VL-7B scores 53.2% on E-VQA, ahead of fine-tuned baselines under the same generator.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

A testable extension: the same descriptor-plus-gating recipe should transfer to other multi-source retrieval problems, such as product search over image and text indexes or fact-checking across heterogeneous document collections, because rank, similarity, and distributional statistics are source-agnostic features.
The gap between the gains on E-VQA (6.7) and InfoSeek (1.2) suggests source complementarity pays off more on larger, noisier knowledge bases; evaluating on the full 6M-page InfoSeek knowledge base instead of the 100k subset is a direct way to check that.
The paper's own failure cases point to a learned source-reliability estimator, the direction its authors suggest, as the natural replacement for Entropy-Weighted Source Fusion to fix the extreme-confidence imbalance where one peaked source overrides correct evidence from the other.
Because the paper reports EWSF ablations only for E-VQA, the InfoSeek gain of 1.2 points is not yet attributable to the fusion weights rather than to the candidate union alone; an InfoSeek ablation isolating gating from entropy-weighted fusion would settle the attribution.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Colleague,

The short version: this is a solid, well-engineered reranking paper for KB-VQA that deserves a serious referee. The new thing is the way it fuses I2I and I2T retrieval sources: a Coarse Retrieval Descriptor (rank, similarity, z-score, gap, entropy) conditions the modality attention weights of a VISTA-style reranker, and an entropy-weighted coarse score is added to the learned score. The whole package — contrastive loss plus auxiliary modality-preserving loss — unifies entity and section retrieval in one 197M model, and the efficiency numbers are credible.

What the paper does well: the ablations are unusually thorough. Table 10 is the most useful part — it shows that on E-VQA, with the same union pool, the base VISTA reranker gets 41.6 R@1, adding RAMG gets 46.7, adding EWSF gets 49.5. That isolates the contribution of the proposed components from the union-pool effect. Table 9 similarly shows VISTA has lower base recall than BLIP-2, so the final gains over EchoSight/OMGM aren't just encoder superiority. The oracle analysis in Figure 3 gives a clean upper bound for each retrieval setting.

The soft spots are real but not fatal. The headline "strongest baselines" is imprecise: Table 1's Δ is against the best single-source baseline (OMGM). The paper never runs a simple heterogeneous-source fusion baseline — RRF or score-sum on the two coarse lists — so the reader can't see whether the +6.7 on E-VQA is more than what a trivial fusion would get from the larger union pool. The internal ablation partly answers this, but an external baseline plus the proposed model on InfoSeek would settle it. Also, no error bars or multi-seed runs; several hyperparameters (η, α, τ_h) are selected on the evaluation splits; and the InfoSeek section pseudo-labels from jina-reranker-v3 are never quantified. These are addressable in revision, not reasons to reject.

If I were the editor, I'd send it out. The method is clearly explained, the claims are mostly supported, and the missing baseline is a straightforward fix. The paper will be useful to anyone working on retrieval-augmented VQA or multimodal reranking.

Referee Report

2 major / 5 minor

Summary. The paper proposes UniHEAR, a lightweight retrieval-reranking framework for knowledge-based VQA. It combines image-to-image (I2I) and image-to-text (I2T) coarse retrieval into a union candidate pool, represents each candidate by a ten-dimensional Coarse Retrieval Descriptor (rank, similarity, z-score, gap, and pool entropy per source), and uses Retrieval-Guided Attentive Modality Gating (RAMG) to condition modality attention on this descriptor. A training-free Entropy-Weighted Source Fusion (EWSF) re-incorporates coarse retrieval scores, and a hybrid contrastive-plus-auxiliary loss unifies entity- and section-level reranking. Experiments on E-VQA and InfoSeek report state-of-the-art Recall@1 (49.5 and 65.2, claimed +6.7 and +1.2 over the strongest baselines) and competitive VQA accuracy with a 197M-parameter reranker, supported by ablations, an oracle analysis, fairness controls, and hyperparameter sweeps.

Significance. If the reported results hold, the paper makes a useful empirical contribution: a 197M-parameter reranker with source-conditioned modality gating outperforms heavier BLIP-2-based rerankers, and the analysis in Figure 3 and Tables 5, 6, 10, and 11 provides unusually thorough ablation coverage. The paper also ships code and model links, and there is no derivational circularity in the main claim: Eq. (13) is an empirical ensemble, and the reported metrics are measurements. The main gap is that the central SOTA claim is not yet tested against a heterogeneous-source fusion baseline, and all reported numbers are single-seed point estimates; these issues are load-bearing for the headline comparisons but appear addressable within the manuscript's experimental scope.

major comments (2)

[§4.3, Tables 1 and 10] The headline +6.7/+1.2 Recall@1 deltas in Table 1 are computed against the best single-source reranker (OMGM on I2T), not against any method that also fuses the I2I and I2T candidate sources. The only matched-pool evidence is the internal 'w/o RAMG&EWSF' row in Table 10 (41.6 on E-VQA), which is an ablation of the proposed VISTA reranker rather than an external fusion baseline, and no equivalent row is reported for InfoSeek. Because the union pool has up to twice the candidates of a single source, the reported gain conflates the benefit of additional candidates with the benefit of RAMG/EWSF. I note that Table 10's matched-pool ablation already weakens the simplest union-pool explanation, but it does not replace a comparison to standard fusion methods. Please add heterogeneous-source fusion baselines (e.g., reciprocal rank fusion, score-sum/weighted score fusion, and a strong existing reranker applied to the union pool) on both E-VQA and InfoSeek, and report the Table 5 component ablation on InfoSeek as well.
[Tables 1 and 2] All retrieval and VQA results are reported as single point estimates without error bars, variance across seeds, or significance tests. Several headline improvements are small — InfoSeek Recall@1 +1.2 in Table 1 and InfoSeek VQA Overall +0.5 over OMGM in Table 2 — and could lie within run-to-run noise for a pipeline with multiple trained components (reranker, auxiliary loss, and generation). Please report means and standard deviations over at least three seeds for the main comparisons and, where feasible, paired significance tests. Without this, the strength of the SOTA claim, especially on InfoSeek, is not robustly supported.

minor comments (5)

[Eq. (13) and Appendix C.3] Eq. (13) presents the final ranking score as an unweighted sum, but Appendix C.3 sweeps a fusion weight eta and adopts eta=0.5. State explicitly in Section 3.2 that Eq. (13) corresponds to eta=0.5, or give the weighted form in the main text, so the main text and appendix are consistent.
[§4.2] The hyperparameter list in Section 4.2 omits eta, the fusion weight introduced in the final score; add eta to the list for reproducibility.
[Table 8] The InfoSeek row labels the 71,335 evaluation samples as 'Test', but Section 4.1 states that evaluation is performed on the validation split; rename the entry to 'Valid' or otherwise clarify the discrepancy.
[Appendix D, Rows 4-5] The case study acknowledges that EWSF can over-weight a high-confidence incorrect source and push the ground-truth entity to rank 20 or 24. Since EWSF is one of the two main contributions, it would be helpful to quantify how often this failure mode occurs (e.g., the fraction of samples where EWSF changes a correct top-1 into an incorrect one) and to state whether the default hyperparameters mitigate it.
[Eq. (3)] The descriptor component z_s(e) is described only as 'z-normalized similarity'; specify the normalization statistics (over which set and whether across the source-specific pool or the union pool) so the construction is unambiguous.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity found: UniHEAR's reported numbers are empirical measurements, and the final ranking score is an explicit ensemble of a learned reranker score and a deliberate coarse-retrieval prior, not a quantity derived from its own output.

full rationale

The paper's derivation chain is self-contained. The final entity score (Eq. 13) is defined as sim(e)=sim_r(e)+sim_c(e), an explicit combination of a learned reranking score and a coarse retrieval prior; using the same coarse similarities in the Coarse Retrieval Descriptor (Eq. 4) and in Entropy-Weighted Source Fusion (Eqs. 11-12) is intentional feature engineering, not a tautology, because the learned sim_r can override the prior and the ablations (Tables 5 and 10) show that each component changes measured Recall@1. The headline Recall@1 and VQA figures are held-out measurements against fixed benchmarks, not derived quantities, and the entropy weights, calibration ratio, and fusion weight are hand-designed or tuned hyperparameters, not fitted parameters renamed as predictions. There are no load-bearing self-citations: the authors do not cite their own prior work, and the cited external components (VISTA, EVA-CLIP, Faiss, and the baselines) provide independent mechanisms. The absence of an external heterogeneous-source fusion baseline in Table 1 is a legitimate benchmarking concern about external validity of the SOTA claim, not circularity of the derivation; likewise, the Appendix D failure cases acknowledge EWSF sensitivity as a limitation, which is empirical rather than circular.

Assumptions & free parameters 7 free parameters · 5 assumptions · 0 invented entities

The paper's central claim depends on several domain-specific heuristics: the complementarity of retrieval sources, the entropy-based reliability weighting, the pseudo-labeling of InfoSeek sections, and the choice of several hyperparameters tuned on evaluation splits. None of these are derived from first principles, so the contribution is empirical and conditional on these assumptions holding on a given KB-VQA benchmark.

free parameters (7)

Entropy temperature tau_h = 0.008
Tuned in Appendix C.3 (Table 13) to maximize discriminability of H_s on E-VQA statistics; assumed to transfer to InfoSeek.
Fusion weight eta = 0.5
Chosen from Figure 5 sweep on evaluation splits; close to per-dataset optima (0.6 on E-VQA, 0.5 on InfoSeek).
Section scoring weight alpha = 0.2
Chosen from Table 12 sweep on GT section coverage; best or near-best across k_s values.
Pool-mean ratio R = data-derived (mu_v/mu_t)
Calibration factor for textual similarities in Eq. 12, computed from candidate pools rather than fixed; no stability analysis across datasets or KB subsets.
Auxiliary loss weight lambda = 0.1
Hand-chosen to balance L_cont and L_aux in Eq. 17.
Mask threshold theta = 0.1
Hand-chosen in Eq. 16 to filter unreliable visual positives; no sensitivity analysis reported.
Coarse retrieval top-k per source = 20
Adopted after Figure 3 tradeoff between recall and latency; per source.

assumptions (5)

domain assumption I2I and I2T coarse retrieval sources are complementary for KB-VQA entities, so the union of top-k candidate sets raises the oracle recall ceiling.
The method's first stage and the entire motivation (Section 1, Figure 3) rest on this. The oracle analysis in Figure 3 supports it for E-VQA, but it is a dataset-specific assumption, not a guarantee.
domain assumption Pool-level entropy H_s over coarse retrieval similarities is a reliable, transferable indicator of source reliability for the EWSF weights.
Used in Eq. 3 and Eq. 11; the paper tunes tau_h to make H_s discriminative on E-VQA but does not verify on InfoSeek.
domain assumption The VISTA encoder with frozen visual tower provides sufficient base representations for reranking; fine-tuning only the text encoder plus RAMG parameters is enough.
The architecture (Section 3.2) inherits VISTA's pretrained space; the paper freezes visual weights (Appendix B).
domain assumption InfoSeek sections pseudo-labeled by jina-reranker-v3 are accurate enough to serve as training positives for the auxiliary loss.
Appendix B states the dataset lacks GT section annotations, so pseudo-labels are used; noise is unquantified.
standard math Standard inner-product similarity and softmax normalization are used throughout (Eqs. 1, 7, 14).
Background math, not a contribution.

how reviews work

0 comments

Cite this review

Pith. "Pith review of UniHEAR: Unified Heterogeneous-Source Attentive Retrieval for Knowledge-Based Visual Question Answering." pith.science (2026). https://pith.science/paper/VGQD7TC3

@misc{pith2026260801147,
  author       = {Pith},
  title        = {Pith review of: UniHEAR: Unified Heterogeneous-Source Attentive Retrieval for Knowledge-Based Visual Question Answering},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/VGQD7TC3}},
  note         = {Machine review of arXiv:2608.01147}
}

read the original abstract

Knowledge-Based Visual Question Answering (KB-VQA) requires retrieving entity knowledge from external sources to answer visually grounded questions. Existing retrieval-augmented systems suffer from two critical limitations. First, relying on a single retrieval modality creates a Single-Source Retrieval Bottleneck, missing ground-truth entities that are only accessible through complementary sources. Second, dual-tower pointwise rerankers suffer from Retrieval-Source-Blind Reranking, as they overlook retrieval origins and candidate-level retrieval priors, leading to redundant modality reliance. To address these challenges, we propose UniHEAR, a unified lightweight framework for heterogeneous-source entity retrieval and reranking. UniHEAR constructs a Coarse Retrieval Descriptor for each candidate entity, and introduces Retrieval-Guided Attentive Modality Gating to condition modality attention weights on this descriptor, complemented by Entropy-Weighted Source Fusion of coarse retrieval priors. A hybrid training strategy combining contrastive learning with an auxiliary modality-preserving loss unifies entity-level and section-level retrieval within a single model. Extensive experiments on E-VQA and InfoSeek demonstrate that UniHEAR achieves state-of-the-art retrieval and VQA performance, improving Recall@1 by 6.7 and 1.2 points over the strongest baselines while maintaining a lightweight reranking architecture. Code and model are available at https://github.com/iven-luo/UniHEAR.

Figures

Figures reproduced from arXiv: 2608.01147 by the authors.

**Figure 1.** Limitations of existing KB-VQA systems. 1 Introduction Visual Question Answering (VQA) [2] requires models to answer natural language questions about images. A more challenging variant, Knowledge-Based VQA (KB-VQA), further requires external entity knowledge beyond visual content from sources like Wikipedia. While Multimodal Large Language Models (MLLMs) [3, 21, 39] arXiv:2608.01147v1 [cs.IR] 2 Aug 2026 [PITH_FULL… view at source ↗

**Figure 2.** Overview of the UniHEAR framework. not merely textual, as it demands joint reasoning over visual content, question semantics, and heterogeneous knowledge sources, making naive similarity-based selection insufficient. A well-designed filtering mechanism therefore serves as an essential bridge between coarse retrieval and precise answer generation, determining not only which knowledge is retrieved, but which knowledg… view at source ↗

**Figure 3.** Analysis of heterogeneous-source retrieval under vary [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗

Figures from the paper (4 more)

**Figure 4.** Figure 4: Effect of the number of reranked entities [PITH_FULL_IMAGE:figures/full_fig_p012_4.png]

**Figure 5.** Figure 5: Effect of the entity similarity fusion weight [PITH_FULL_IMAGE:figures/full_fig_p013_5.png]

**Figure 5.** Figure 5: Effect of the entity similarity fusion weight [PITH_FULL_IMAGE:figures/full_fig_p012_5.png]

**Figure 6.** Figure 6: Qualitative retrieval examples on E-VQA and InfoSeek. Each sample shows the query and retrieval results with ground-truth [PITH_FULL_IMAGE:figures/full_fig_p014_6.png]

Review history (2 revisions) →

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

40 extracted references · 6 canonical work pages

[1]

Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Floren- cia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. 2023. Gpt-4 technical report.arXiv preprint arXiv:2303.08774 (2023)

arXiv 2023
[2]

Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Margaret Mitchell, Dhruv Batra, C Lawrence Zitnick, and Devi Parikh. 2015. Vqa: Visual question answering. In Proceedings of the IEEE/CVF International Conference on Computer Vision. 2425– 2433

work page 2015
[3]

Shuai Bai, Yuxuan Cai, Ruizhe Chen, Keqin Chen, Xionghui Chen, Zesen Cheng, Lianghao Deng, Wei Ding, Chang Gao, Chunjiang Ge, Wenbin Ge, Zhifang Guo, Qidong Huang, Jie Huang, Fei Huang, Binyuan Hui, Shutong Jiang, Zhaohai Li, Mingsheng Li, Mei Li, Kaixin Li, Zicheng Lin, Junyang Lin, Xuejing Liu, Jiawei Liu, Chenglong Liu, Yang Liu, Dayiheng Liu, Shixuan ...

arXiv 2025
[4]

Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, Humen Zhong, Yuanzhi Zhu, Mingkun Yang, Zhaohai Li, Jianqiang Wan, Pengfei Wang, Wei Ding, Zheren Fu, Yiheng Xu, Jiabo Ye, Xi Zhang, Tianbao Xie, Zesen Cheng, Hang Zhang, Zhibo Yang, Haiyang Xu, and Junyang Lin. 2025. Qwen2.5-VL Technical Rep...

arXiv 2025
[5]

Davide Caffagni, Federico Cocchi, Nicholas Moratelli, Sara Sarto, Marcella Cornia, Lorenzo Baraldi, and Rita Cucchiara. 2024. Wiki-llava: Hierarchical retrieval- augmented generation for multimodal llms. InProceedings of the IEEE/CVF Con- ference on Computer Vision and Pattern Recognition. 1818–1826

2024
[6]

Jianlyu Chen, Shitao Xiao, Peitian Zhang, Kun Luo, Defu Lian, and Zheng Liu
[7]

Yang Chen, Hexiang Hu, Yi Luan, Haitian Sun, Soravit Changpinyo, Alan Ritter, and Ming-Wei Chang. 2023. Can pre-trained vision and language models answer visual information-seeking questions?arXiv preprint arXiv:2302.11713(2023)

arXiv 2023
[8]

Federico Cocchi, Nicholas Moratelli, Marcella Cornia, Lorenzo Baraldi, and Rita Cucchiara. 2025. Augmenting multimodal llms with self-reflective tokens for knowledge-based visual question answering. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 9199–9209

2025

Show all 40 references

[9]

Alberto Compagnoni, Marco Morini, Sara Sarto, Federico Cocchi, Davide Caffagni, Marcella Cornia, Lorenzo Baraldi, and Rita Cucchiara. 2025. ReAG: Reasoning- Augmented Generation for Knowledge-based Visual Question Answering.arXiv preprint arXiv:2511.22715(2025)

2025
[10]

Lianghao Deng, Yuchong Sun, Shizhe Chen, Ning Yang, Yunfeng Wang, and Ruihua Song. 2025. MuKA: Multimodal knowledge augmented visual information- seeking. InProceedings of the 31st International Conference on Computational Linguistics. 9675–9686

2025
[11]

Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 conference of the North American chapter of the association for computational linguistics: human...

2019
[12]

Alexey Dosovitskiy. 2020. An image is worth 16x16 words: Transformers for image recognition at scale.arXiv preprint arXiv:2010.11929(2020)

2020 arXiv
[13]

Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. 2024. The llama 3 herd of models.arXiv preprint arXiv:2407.21783 (2024)

2024 arXiv
[14]

Yuyang Hong, Jiaqi Gu, Qi Yang, Lubin Fan, Yue Wu, Ying Wang, Kun Ding, Shim- ing Xiang, and Jieping Ye. 2025. Knowledge-based Visual Question Answer with Multimodal Processing, Retrieval and Filtering.arXiv preprint arXiv:2510.14605 (2025)

2025
[15]

Yushi Hu, Hang Hua, Zhengyuan Yang, Weijia Shi, Noah A Smith, and Jiebo Luo. 2023. Promptcap: Prompt-guided image captioning for vqa with gpt-3. In Proceedings of the IEEE/CVF International Conference on Computer Vision. 2963– 2975

2023
[16]

Pu Jian, Donglei Yu, and Jiajun Zhang. 2024. Large language models know what is key visual entity: An LLM-assisted multimodal retrieval for VQA. InProceedings of the 2024 Conference on Empirical Methods in Natural Language Processing. 10939–10956

2024
[17]

Jeff Johnson, Matthijs Douze, and Hervé Jégou. 2019. Billion-scale similarity search with GPUs.IEEE Transactions on Big Data7, 3 (2019), 535–547

2019
[18]

Mahmoud Khademi, Ziyi Yang, Felipe Frujeri, and Chenguang Zhu. 2023. MM- reasoner: A multi-modal knowledge-aware framework for knowledge-based visual question answering. InFindings of the Association for Computational Lin- guistics: EMNLP 2023. 6571–6581

2023
[19]

Weizhe Lin and Bill Byrne. 2022. Retrieval augmented visual question answering with outside knowledge.arXiv preprint arXiv:2210.03809(2022)

2022 arXiv
[20]

Weizhe Lin, Jingbiao Mei, Jinghong Chen, and Bill Byrne. 2024. Preflmr: Scal- ing up fine-grained late-interaction multi-modal retrievers.arXiv preprint arXiv:2402.08327(2024)

2024 arXiv
[21]

Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. 2024. Improved baselines with visual instruction tuning. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 26296–26306

2024
[22]

Thomas Mensink, Jasper Uijlings, Lluis Castrejon, Arushi Goel, Felipe Cadar, Howard Zhou, Fei Sha, André Araujo, and Vittorio Ferrari. 2023. Encyclopedic vqa: Visual questions about detailed properties of fine-grained categories. In Proceedings of the IEEE/CVF International Co...

2023
[23]

Nitesh Methani, Pritha Ganguly, Mitesh M Khapra, and Pratyush Kumar. 2020. Plotqa: Reasoning over scientific plots. InProceedings of the ieee/cvf winter con- ference on applications of computer vision. 1527–1536

2020
[24]

Shan Ning, Longtian Qiu, and Xuming He. 2026. Wiki-R1: Incentivizing Multi- modal Reasoning for Knowledge-based VQA via Data and Sampling Curriculum. arXiv preprint arXiv:2603.05256(2026)

2026 arXiv
[25]

Jingyuan Qi, Zhiyang Xu, Rulin Shao, Yang Chen, Jin Di, Yu Cheng, Qifan Wang, and Lifu Huang. 2024. Rora-vlm: Robust retrieval-augmented vision language models.arXiv preprint arXiv:2410.08876(2024)

2024 arXiv
[26]

Zhenwei Shao, Zhou Yu, Meng Wang, and Jun Yu. 2023. Prompting large language models with answer heuristics for knowledge-based visual question answering. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 14974–14983

2023
[27]

Quan Sun, Yuxin Fang, Ledell Wu, Xinlong Wang, and Yue Cao. 2023. Eva-clip: Improved training techniques for clip at scale.arXiv preprint arXiv:2303.15389 (2023)

2023 arXiv
[28]

Quan Sun, Jinsheng Wang, Qiying Yu, Yufeng Cui, Fan Zhang, Xiaosong Zhang, and Xinlong Wang. 2024. Eva-clip-18b: Scaling clip to 18 billion parameters. arXiv preprint arXiv:2402.04252(2024)

2024 arXiv
[29]

Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yas- mine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhos- ale, et al. 2023. Llama 2: Open foundation and fine-tuned chat models.arXiv preprint arXiv:2307.09288(2023)

2023 arXiv
[30]

Feng Wang, Yuqing Li, and Han Xiao. 2025. jina-reranker-v3: Last but Not Late Interaction for Listwise Document Reranking. arXiv:2509.25085 [cs.CL]

2025
[31]

Shitao Xiao, Zheng Liu, Peitian Zhang, Niklas Muennighoff, Defu Lian, and Jian-Yun Nie. 2024. C-pack: Packed resources for general chinese embeddings. InProceedings of the 47th international ACM SIGIR conference on research and development in information retrieval. 641–649

2024
[32]

Yibin Yan and Weidi Xie. 2024. EchoSight: Advancing visual-language models with Wiki knowledge.arXiv preprint arXiv:2407.12735(2024)

2024 arXiv
[33]

Wei Yang, Jingjing Fu, Rui Wang, Jinyu Wang, Lei Song, and Jiang Bian. 2025. OMGM: Orchestrate Multiple Granularities and Modalities for Efficient Multi- modal Retrieval.arXiv preprint arXiv:2505.07879(2025)

2025 arXiv
[34]

Zhengyuan Yang, Zhe Gan, Jianfeng Wang, Xiaowei Hu, Yumao Lu, Zicheng Liu, and Lijuan Wang. 2022. An empirical study of gpt-3 for few-shot knowledge- based vqa. InProceedings of the AAAI conference on artificial intelligence, Vol. 36. 3081–3089

2022
[35]

Xu Yuan, Liangbo Ning, Wenqi Fan, and Qing Li. 2025. mKG-RAG: Multimodal Knowledge Graph-Enhanced RAG for Visual Question Answering.arXiv preprint arXiv:2508.05318(2025)

2025 arXiv
[36]

Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q Weinberger, and Yoav Artzi. 2019. Bertscore: Evaluating text generation with bert.arXiv preprint arXiv:1904.09675(2019)

2019 arXiv
[37]

Tao Zhang, Ziqi Zhang, Zongyang Ma, Yuxin Chen, Zhongang Qi, Chunfeng Yuan, Bing Li, Junfu Pu, Yuxuan Zhao, Zehua Xie, et al. 2024. mR 2 AG: Multimodal Retrieval-Reflection-Augmented Generation for Knowledge-Based VQA.arXiv preprint arXiv:2411.15041(2024)

2024 arXiv
[38]

Junjie Zhou, Zheng Liu, Shitao Xiao, Bo Zhao, and Yongping Xiong. 2024. Vista: Visualized text embedding for universal multi-modal retrieval.arXiv preprint arXiv:2406.04292(2024)

2024 arXiv
[39]

Jinguo Zhu, Weiyun Wang, Zhe Chen, Zhaoyang Liu, Shenglong Ye, Lixin Gu, Hao Tian, Yuchen Duan, Weijie Su, Jie Shao, et al. 2025. Internvl3: Exploring advanced training and test-time recipes for open-source multimodal models. arXiv preprint arXiv:2504.10479(2025). MM ’26, Nove...

2025 arXiv
[2024]

InFindings of the association for computational linguistics: ACL 2024

M3-embedding: Multi-linguality, multi-functionality, multi-granularity text embeddings through self-knowledge distillation. InFindings of the association for computational linguistics: ACL 2024. 2318–2335

2024

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.

[1] [1]

Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Floren- cia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. 2023. Gpt-4 technical report.arXiv preprint arXiv:2303.08774 (2023)

arXiv 2023

[2] [2]

Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Margaret Mitchell, Dhruv Batra, C Lawrence Zitnick, and Devi Parikh. 2015. Vqa: Visual question answering. In Proceedings of the IEEE/CVF International Conference on Computer Vision. 2425– 2433

work page 2015

[3] [3]

Shuai Bai, Yuxuan Cai, Ruizhe Chen, Keqin Chen, Xionghui Chen, Zesen Cheng, Lianghao Deng, Wei Ding, Chang Gao, Chunjiang Ge, Wenbin Ge, Zhifang Guo, Qidong Huang, Jie Huang, Fei Huang, Binyuan Hui, Shutong Jiang, Zhaohai Li, Mingsheng Li, Mei Li, Kaixin Li, Zicheng Lin, Junyang Lin, Xuejing Liu, Jiawei Liu, Chenglong Liu, Yang Liu, Dayiheng Liu, Shixuan ...

arXiv 2025

[4] [4]

Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, Humen Zhong, Yuanzhi Zhu, Mingkun Yang, Zhaohai Li, Jianqiang Wan, Pengfei Wang, Wei Ding, Zheren Fu, Yiheng Xu, Jiabo Ye, Xi Zhang, Tianbao Xie, Zesen Cheng, Hang Zhang, Zhibo Yang, Haiyang Xu, and Junyang Lin. 2025. Qwen2.5-VL Technical Rep...

arXiv 2025

[5] [5]

Davide Caffagni, Federico Cocchi, Nicholas Moratelli, Sara Sarto, Marcella Cornia, Lorenzo Baraldi, and Rita Cucchiara. 2024. Wiki-llava: Hierarchical retrieval- augmented generation for multimodal llms. InProceedings of the IEEE/CVF Con- ference on Computer Vision and Pattern Recognition. 1818–1826

2024

[6] [6]

Jianlyu Chen, Shitao Xiao, Peitian Zhang, Kun Luo, Defu Lian, and Zheng Liu

[7] [7]

Yang Chen, Hexiang Hu, Yi Luan, Haitian Sun, Soravit Changpinyo, Alan Ritter, and Ming-Wei Chang. 2023. Can pre-trained vision and language models answer visual information-seeking questions?arXiv preprint arXiv:2302.11713(2023)

arXiv 2023

[8] [8]

Federico Cocchi, Nicholas Moratelli, Marcella Cornia, Lorenzo Baraldi, and Rita Cucchiara. 2025. Augmenting multimodal llms with self-reflective tokens for knowledge-based visual question answering. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 9199–9209

2025