Pith. sign in

REVIEW 5 major objections 4 minor 41 references

Replacing pooled visual ranking with late-interaction MaxSim scoring in multimodal GraphRAG changes which graph-linked images reach downstream reasoning and improves answers where visual evidence matters.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-02 14:29 UTC pith:CECCWOCW

load-bearing objection Modest, honest swap-in study of late-interaction MaxSim for graph-linked image ranking; the central attribution is plausible but unverified because the visual encoder is never confirmed fixed. the 5 major comments →

arxiv 2607.16208 v1 pith:CECCWOCW submitted 2026-05-09 cs.AI cs.CL

ColGraphRAG: Late-Interaction Evidence Retrieval for Multimodal GraphRAG

Seonok Kim This is my paper
classification cs.AI cs.CL
keywords late-interaction retrievalmultimodal GraphRAGvisual candidate rankingMaxSim scoringevidence graphmultimodal question answeringevidence inclusionretrieval-to-graph bottleneck
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

This paper isolates one bottleneck in graph-grounded multimodal question answering: the way images linked to the evidence graph are ranked before they can be used. It claims that replacing single-vector pooled image similarity with late-interaction MaxSim scoring—where each query token can match the best local image patch—changes which images enter downstream reasoning and improves answer quality where visual evidence matters. The authors keep the graph, text/table retrieval, extraction, and generator unchanged, so any difference in results is attributed to the visual ranking step. On the MultimodalQA benchmark, retrieval-stage rank metrics and aggregate exact-match/F1 point estimates improve, with larger gains on image- and table-grounded questions and mixed (slightly negative) movement on text-dominant ones. The paper presents this as mechanism-level evidence for an evidence-inclusion bottleneck, not a full-benchmark claim.

Core claim

The central claim is that the visual candidate-ranking operator at the boundary between a constructed graph and downstream reasoning is a bottleneck for graph-linked image evidence. A graph may contain the right image, but if a pooled bi-encoder collapses each image to one vector and misses fine-grained query–patch matches, that image can be ranked too low to enter the evidence set. ColGraphRAG replaces only this operator with late-interaction MaxSim scoring, which keeps multiple query vectors and local image units and scores each query token by its best local match. With everything else fixed, this swap is associated with improved retrieval-stage ranking of graph-linked candidates and impro

What carries the argument

The key mechanism is late-interaction MaxSim scoring over graph-linked image candidates. For a question, a deterministic template-based phrase constructor produces a set of visual query phrases; these are encoded into multiple query token vectors Qv. Each candidate image x is represented by local units (patches or regions) U(x). The score is smv(Qv,x) = sum over query tokens of the maximum cosine similarity to any local unit. Deferring pooling keeps local alignment until scoring, so different query tokens can match different regions of the same image. Only this scoring operator differs from the baseline; graph construction, text/table retrieval, extraction, and answer generation are shared.

Load-bearing premise

The deterministic template-based visual phrase constructor (ImagePhrases) is assumed to produce query tokens that align semantically with relevant image patches; the paper provides no template definitions, examples, or encoder details, so if these phrases are not well-aligned, the MaxSim comparison would test template quality rather than late interaction.

What would settle it

Concrete test: run the same pooled-vs-MaxSim comparison but replace the deterministic visual phrases with random query tokens or with phrases from a fixed generic vocabulary. If the MaxSim gains over pooled ranking disappear or reverse, the reported improvement is caused by the phrase templates, not by late-interaction scoring. Alternatively, if pooled scoring given the same well-crafted phrases matches MaxSim performance, then pooling was not the bottleneck.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • In GraphRAG pipelines where images are linked to a graph, the ranking operator that promotes image evidence is a controllable point of intervention; changing it alone can shift downstream answers.
  • Gains concentrate where answers depend on fine-grained visual cues, suggesting retrieval-stage image ranking is a bottleneck for visual-heavy questions.
  • Late-interaction scoring for graph-linked images can be dropped into existing graph-grounded QA stacks without retraining the graph or changing other retrievers.
  • Text-dominant subsets can regress when image ranking changes, indicating that altering the visual gate perturbs coupled extraction even when text evidence is already sufficient.
  • On retrieval-native benchmarks, the same scoring interface achieves strong NDCG@10, so the mechanism appears to transfer beyond the QA stack to visual document retrieval.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The paper's design isolates the scorer but not the phrase constructor; a natural next test is to ablate the deterministic visual phrases (e.g., random phrases vs. curated templates) to confirm that late interaction, not template content, drives the change.
  • The mechanism suggests a general design principle: in multimodal retrieval-augmented generation, the granularity of the similarity operator (pooled vs. token-to-patch) can be as important as the graph structure itself.
  • One could test the claim on other benchmarks by holding the graph and downstream stack fixed and toggling only the image scorer; if gains do not transfer to datasets with different visual evidence distributions, the effect may be dataset-specific.
  • The paper's causal interpretation would be strengthened by tracking whether MaxSim-promoted images actually survive extraction and appear in the final reasoning graph, which the paper lists as future work.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

5 major / 4 minor

Summary. The paper proposes ColGraphRAG, a modification to a multimodal GraphRAG pipeline that replaces the pooled single-vector visual candidate ranking over graph-linked image nodes with late-interaction MaxSim-style multi-vector scoring. The authors report that on MultimodalQA this swap is associated with improved retrieval-stage point estimates (Hit@1, MRR, nDCG) and higher aggregate EM/F1, with larger gains on image- and table-tagged questions and mixed results on text-dominant ones. They also report strongly positive retrieval-native results on ViDoRe v3. The paper explicitly disclaims statistical significance and frames the results as 'mechanism-level evidence' that finer-grained visual ranking improves graph-linked evidence inclusion.

Significance. If the causal interpretation were supported, the paper would provide a clean ablation of the scoring operator within a fixed graph-grounded pipeline, isolating an evidence-inclusion bottleneck that is plausible and practically important. The methodological choices of restricting late interaction to graph-linked images, using deterministic template-based query phrases, and sharing the graph snapshot and non-visual stages are commendable and make the comparison less prone to many common confounds. However, the current evidence is insufficient: the key comparison may confound encoder identity with scoring aggregation, and the lack of any statistical characterization or implementation details for the query-phrase templates undermines reproducibility and the causal chain. The paper is honest about its limitations, but the central claim overreaches relative to what the experiments can support.

major comments (5)
  1. [§3.3, Eq. (1); §4.2] The causal attribution that 'only the visual candidate-ranking operator' changes is not supported because the manuscript never specifies whether the pooled baseline and the MaxSim variant use the same visual encoder. If f(·) in the baseline is a CLIP-style bi-encoder and Enc(·)/U(·) in the proposed method is a ColPali-style late-interaction encoder, the comparison conflates encoder capacity with scoring aggregation. Please state the exact encoder architectures and, ideally, run a pooled baseline obtained by mean-pooling the same late-interaction encoder's token vectors.
  2. [§3.1, Algorithm 1] Cvis(q) = ImagePhrases(q, κ(q), G) is never defined: no template strings, no extraction of κ(q), no encoder identity. This is the sole query-side input to the visual ranking, so improvements could reflect the template quality rather than late interaction. Provide the full template set and an ablation with trivial templates (e.g., just the raw question text) to separate the phrase-construction effect.
  3. [Tables 1, 3, 4] All MultimodalQA results are single point estimates with no error bars, repeated-seed runs, or significance tests. The authors explicitly disclaim statistical significance, yet the paper's central claim is a causal chain (MaxSim ⇒ Ek(q) ⇒ R(q) ⇒ G⋆(q) ⇒ â). Please report at least 5 seeds with mean ± std, and a paired test (e.g., Wilcoxon signed-rank) for the key comparisons. Without this, 'mechanism-level evidence' overstates the support.
  4. [Table 2, §4.5] The ColGraphRAG row on ViDoRe v3 achieves 70.2 average NDCG@10, far above all baselines (best baseline 61.4). It is unclear what is actually evaluated: the method is a graph-based QA pipeline, not a document retriever. Specify the exact task (retrieval of pages? images?), the encoder, the index, and any use of the graph; otherwise the result is not interpretable and may be an artifact of a different evaluation protocol.
  5. [§4.4] The chain 'MaxSim ⇒ Ek(q) ⇒ R(q) ⇒ G⋆(q) ⇒ â' is asserted but the intermediate step that improved ranking leads to more gold evidence surviving extraction is not measured. The authors acknowledge this ('finer tabulations... future work'), but the main claim rests on this link. Add direct measurements: among questions where Ek(q) changes, how often does the gold image enter R(q)? Report a confusion-style breakdown.
minor comments (4)
  1. [Figure 4] The axis labels appear corrupted by Unicode escapes (/uni00000014 etc.); replace with proper text.
  2. [§4.3 and Table 2] 'nDCG@k' vs 'NDCG@10' have inconsistent capitalization; use one style throughout.
  3. [Figure 2] The phrase 'eatre apping ayer' appears garbled; proofread the caption and the figure text.
  4. [Related work] The related work is thin on recent graph-based multimodal RAG systems; a few more 2024–2025 citations would help position the contribution.

Circularity Check

0 steps flagged

No significant circularity: the comparison is an empirical ablation with no fitted parameters and no self-citation chain.

full rationale

The paper's central claim is an empirical comparison between two visual ranking operators over the same graph-linked image pool, holding all other pipeline components fixed. There are no fitted parameters: the scoring functions are defined deterministically (Eq. 1) and the visual phrase constructor Cvis(q) is described as template-based and non-generative, with no indication that the templates or κ(q) depend on the gold answers or target outputs. The retrieval and QA numbers are point estimates from a fixed evaluation protocol, not quantities derived from the same data via fitting. No load-bearing step is a renamed input, a fitted parameter called a prediction, or an imported uniqueness theorem. The paper explicitly scopes its claims and notes the lack of statistical significance and the need for finer diagnostics (Section 5 and the interpretation paragraph in Section 4.4). Potential concerns identified by the reviewer, such as a possible encoder mismatch between the pooled baseline and the MaxSim variant, are external validity or confounding issues, not circular reasoning, because the paper does not define the baseline in terms of the MaxSim result. Likewise, the reliance on deterministic templates is a test of template quality, but the paper does not use the target labels to define the templates. No self-citations are present; the only related work citations are independent sources. The ViDoRe v3 evaluation is an external benchmark, providing independent evidence rather than circular support. Therefore, the derivation chain is self-contained with respect to circularity, and no specific reduction can be exhibited.

Axiom & Free-Parameter Ledger

2 free parameters · 4 axioms · 0 invented entities

The paper introduces no new entities or fitted mathematical constants. Its main hidden inputs are the undisclosed visual phrase templates, the undisclosed graph snapshot, and the unstated hyperparameters; these are hand-chosen design elements rather than numerically fitted parameters.

free parameters (2)
  • Visual query phrase templates ImagePhrases(·) = not specified
    Cvis(q) = ImagePhrases(q, κ(q), G) is deterministic but the templates themselves are never defined; the success of the visual branch depends on them.
  • Candidate budget k and expansion rounds T = held constant, values not reported
    k gates how many images enter extraction and T gates graph expansion; both are load-bearing for the end-task comparison but their values are not disclosed.
axioms (4)
  • domain assumption The pre-built graph G already links relevant image nodes to entities via r_img edges, so Ximg(G) contains the gold images.
    Invoked throughout Section 3.1 and Algorithm 1; if graph construction omits relevant images, no image ranker can recover them. The paper does not evaluate graph-construction recall.
  • domain assumption Query-side visual tokens Qv = Enc(Cvis(q)) and local visual units U(x) live in the same retrieval space with a meaningful similarity.
    Equation 1 assumes a shared embedding space; the paper does not specify the encoders or alignment, only cites the ColPali/ColBERT lineage.
  • ad hoc to paper The deterministic phrase templates ImagePhrases(q, κ(q), G) produce semantically adequate query phrases.
    Cvis(q) definition in Section 3.1 is load-bearing but no template contents or examples are provided.
  • domain assumption End-to-end differences can be attributed to the visual ranker swap because all other stages are unchanged.
    Section 4.1 states this attribution; but extraction coupling could be affected by changes in Ek(q), and the paper acknowledges coupled extraction effects in Section 4.4.

pith-pipeline@v1.3.0-alltime-deepseek · 13132 in / 13004 out tokens · 144993 ms · 2026-08-02T14:29:58.976784+00:00 · methodology

0 comments
read the original abstract

Graph-grounded multimodal question answering organizes text, tables, and images in a structured evidence graph, yet end-to-end accuracy depends on which multimodal assets are ranked highly enough to enter downstream reasoning; for graph-linked images, single-vector bi-encoder similarity can discard patch- and token-level structure needed for fine-grained alignment. We evaluate replacing the visual candidate-ranking operator over graph-linked image nodes with late-interaction MaxSim-style multi-vector scoring in the ColBERT/ColPali lineage, while keeping offline graph construction, text- and table-side retrieval, structured extraction, and downstream reasoning unchanged. On MultimodalQA, this change is associated with improved retrieval-stage point estimates for graph-linked image candidates and downstream QA gains, with larger movement where visual evidence matters most and mixed trends on text-dominant questions; we interpret the pattern as mechanism-level evidence for graph-linked visual evidence inclusion, while broader validation and finer graph-level diagnostics remain important future work.

Figures

Figures reproduced from arXiv: 2607.16208 by Seonok Kim.

Figure 1
Figure 1. Figure 1: Overview of ColGraphRAG. Given a multimodal knowledge base, ColGraphRAG derives [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Illustrative case study of evidence linking and graph-based reasoning. For the query asking [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Single-vector pooled ranking versus late-interaction MaxSim candidate ranking. Both [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: ViDoRe v3 subsets: retrieval performance across top- [PITH_FULL_IMAGE:figures/full_fig_p013_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Qualitative MultimodalQA examples. Each panel shows a question, gold evidence, a [PITH_FULL_IMAGE:figures/full_fig_p014_5.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

41 extracted references · 1 canonical work pages

  1. [1]

    Talmor, O

    A. Talmor, O. Yoran, A. Catav, D. Lahav, Y . Wang, A. Asai, G. Ilharco, H. Hajishirzi, and J. Berant. MultiModalQA: Complex Question Answering over Text, Tables and Images. InInternational Conference on Learning Representations (ICLR), 2021

    2021

  2. [2]

    M. Suri, P. Mathur, F. Dernoncourt, K. Goswami, R. A. Rossi, and D. Manocha. VisDoM: Multi-Document QA with Visually Rich Elements Using Multimodal Retrieval-Augmented Generation. InProceedings of NAACL, 2025

    2025

  3. [3]

    Tanaka, T

    R. Tanaka, T. Iki, T. Hasegawa, K. Nishida, K. Saito, and J. Suzuki. VDocRAG: Retrieval-Augmented Generation over Visually-Rich Documents. InProceedings of CVPR, 2025

    2025

  4. [4]

    Y . Ma, Y . Zang, L. Chen, M. Chen, Y . Jiao, X. Li, X. Lu, Z. Liu, Y . Ma, X. Dong, P. Zhang, L. Pan, Y .-G. Jiang, J. Wang, Y . Cao, and A. Sun. MMLongBench-Doc: Benchmarking Long-context Document Understanding with Visualizations. InAdvances in Neural Information Processing Systems, 2024

    2024

  5. [5]

    C. Deng, J. Yuan, P. Bu, P. Wang, Z.-Z. Li, J. Xu, X.-H. Li, Y . Gao, J. Song, B. Zheng, and C.-L. Liu. LongDocURL: a Comprehensive Multimodal Long Document Benchmark Integrating Understanding, Reasoning, and Locating. InProceedings of ACL (Long Papers), 2025

    2025

  6. [6]

    Z. Yang, P. Qi, S. Zhang, Y . Bengio, W. Cohen, R. Salakhutdinov, and C. D. Manning. HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answering. InProceedings of EMNLP, 2018

    2018

  7. [7]

    Trivedi, N

    H. Trivedi, N. Balasubramanian, T. Khot, and A. Sabharwal. MuSiQue: Multihop Questions via Single-hop Question Composition.Transactions of the Association for Computational Linguistics, 10:539–554, 2022

    2022

  8. [8]

    He and X

    X. He and X. Wang. Multimodal Graph Transformer for Multimodal Question Answering. InProceedings of EACL, 2023

    2023

  9. [9]

    C. Bu, G. Chang, Z. Chen, C. Dang, Z. Wu, Y . He, and X. Wu. Query-Driven Multimodal GraphRAG: Dynamic Local Knowledge Graph Construction for Online Reasoning. InFindings of the Association for Computational Linguistics: ACL 2025, pages 21360–21380, Vienna, Austria. Association for Compu- tational Linguistics. DOI: 10.18653/v1/2025.findings-acl.1100. URL:...

    work page doi:10.18653/v1/2025.findings-acl.1100 2025

  10. [10]

    Chang, M

    Y . Chang, M. Narang, H. Suzuki, G. Cao, J. Gao, and Y . Bisk. WebQA: Multihop and Multimodal Question Answering. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 16495–16504, 2022

    2022

  11. [11]

    Yang, D.-K

    C. Yang, D.-K. Vu, M.-T. Nguyen, X.-Q. Nguyen, L. Nguyen, and H. Le. SuperRAG: Beyond RAG with Layout-Aware Graph Modeling. InProceedings of NAACL (Industry Track), 2025

    2025

  12. [12]

    Radford, J

    A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark, G. Krueger, and I. Sutskever. Learning Transferable Visual Models From Natural Language Supervision. InProceedings of ICML, 2021

    2021

  13. [13]

    Khattab and M

    O. Khattab and M. Zaharia. ColBERT: Efficient and Effective Passage Search via Contextualized Late Interaction over BERT. InProceedings of SIGIR, 2020

    2020

  14. [14]

    Santhanam, O

    K. Santhanam, O. Khattab, J. Saad-Falcon, C. Potts, and M. Zaharia. ColBERTv2: Effective and Efficient Retrieval via Lightweight Late Interaction. InProceedings of NAACL, 2022

    2022

  15. [15]

    Faysse, H

    M. Faysse, H. Sibille, T. Wu, B. Omrani, G. Viaud, C. Hudelot, and P. Colombo. ColPali: Efficient Docu- ment Retrieval with Vision Language Models. InInternational Conference on Learning Representations (ICLR), 2025

    2025

  16. [16]

    W. Lin, J. Chen, J. Mei, A. Coca, and B. Byrne. Fine-grained Late-interaction Multi-modal Retrieval for Retrieval Augmented Visual Question Answering. InAdvances in Neural Information Processing Systems, 2023

    2023

  17. [17]

    W. Lin, J. Mei, J. Chen, and B. Byrne. PreFLMR: Scaling Up Fine-Grained Late-Interaction Multi-modal Retrievers. InProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 2024. 10

    2024

  18. [18]

    K. Dong, Y . Chang, D. Goh, D. Li, R. Tang, and Y . Liu. MMDocIR: Benchmarking Multimodal Retrieval for Long Documents. InProceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, 2025

    2025

  19. [19]

    Wasserman, R

    N. Wasserman, R. Pony, O. Naparstek, A. R. Goldfarb, E. Schwartz, U. Barzelay, and L. Karlinsky. REAL-MM-RAG: A Real-World Multi-Modal Retrieval Benchmark. InProceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 2025

    2025

  20. [20]

    Y . K. Chia, L. Cheng, H. P. Chan, M. Song, C. Liu, M. Aljunied, S. Poria, and L. Bing. M-LongDoc: A Benchmark For Multimodal Super-Long Document Understanding And A Retrieval-Aware Tuning Framework. InProceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, 2025

    2025

  21. [21]

    C. Jain, Y . Wu, Y . Zeng, J. Liu, S. Dai, Z. Shao, Q. Wu, and H. Wang. SimpleDoc: Multi-Modal Document Understanding with Dual-Cue Page Retrieval and Iterative Refinement. InProceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, 2025

    2025

  22. [22]

    Van Landeghem, R

    J. Van Landeghem, R. Tito, Ł. Borchmann, M. Pietruszka, P. Joziak, R. Powalski, D. Jurkiewicz, M. Cous- taty, B. Anckaert, E. Valveny, M. Blaschko, S. Moens, and T. Stanislawek. Document Understanding Dataset and Evaluation (DUDE). InProceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), 2023

    2023

  23. [23]

    T. Blau, S. Fogel, R. Ronen, A. Golts, R. Ganz, E. Ben Avraham, A. Aberdam, S. Tsiper, and R. Litman. GRAM: Global Reasoning for Multi-Page VQA. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2024

    2024

  24. [24]

    J. Chen, S. Xiao, P. Zhang, K. Luo, D. Lian, and Z. Liu. BGE M3-Embedding: Multi-lingual, multi- functionality, multi-granularity text embeddings through self-knowledge distillation.arXiv preprint arXiv:2402.03216, 2024

    Pith/arXiv arXiv 2024

  25. [25]

    Günther, S

    M. Günther, S. Sturua, M. K. Akram, I. Mohr, A. Ungureanu, S. Eslami, S. Martens, B. Wang, N. Wang, and H. Xiao. jina-embeddings-v4: Universal embeddings for multimodal multilingual retrieval.arXiv preprint arXiv:2506.18902, 2025

    Pith/arXiv arXiv 2025

  26. [26]

    Zhang, M

    Y . Zhang, M. Li, D. Long, X. Zhang, H. Lin, B. Yang, P. Xie, A. Yang, D. Liu, J. Lin, F. Huang, and J. Zhou. Qwen3 Embedding: Advancing text embedding and reranking through foundation models.arXiv preprint arXiv:2506.05176, 2025

    Pith/arXiv arXiv 2025

  27. [27]

    S. Bai, K. Chen, X. Liu, J. Wang, W. Ge, S. Song, K. Dang, P. Wang, S. Wang, J. Tang, et al. Qwen2.5-VL technical report.arXiv preprint arXiv:2502.13923, 2025

    Pith/arXiv arXiv 2025

  28. [28]

    Nomic Embed Multimodal: Interleaved text, image, and screenshots for vi- sual document retrieval

    Nomic Team. Nomic Embed Multimodal: Interleaved text, image, and screenshots for vi- sual document retrieval. Nomic AI blog, 2025. Available at https://nomic.ai/blog/posts/ nomic-embed-multimodal

    2025

  29. [29]

    M. Xu, G. Moreira, R. Ak, R. Osmulski, Y . Babakhin, Z. Yu, B. Schifferer, and E. Oldridge. Llama Nemoretriever Colembed: Top-performing text-image retrieval model.arXiv preprint arXiv:2507.05513, 2025

    Pith/arXiv arXiv 2025

  30. [30]

    G. d. S. P. Moreira, R. Ak, M. Xu, O. Holworthy, B. Schifferer, Z. Yu, Y . Babakhin, R. Osmulski, J. Cai, R. Chesler, B. Liu, and E. Oldridge. Nemotron ColEmbed V2: Top-performing late interaction embedding models for visual document retrieval.arXiv preprint arXiv:2602.03992, 2026

    arXiv 2026

  31. [31]

    W. Liu, F. Lei, T. Luo, J. Lei, S. He, J. Zhao, and K. Liu. MMHQA-ICL: Multimodal in-context learning for hybrid question answering over text, tables and images.arXiv preprint arXiv:2309.04790, 2023

    Pith/arXiv arXiv 2023

  32. [32]

    W. Chen, H. Hu, X. Chen, P. Verga, and W. Cohen. MuRAG: Multimodal retrieval-augmented generator for open question answering over images and text. InProceedings of the Conference on Empirical Methods in Natural Language Processing, pages 5558–5570, 2022

    2022

  33. [33]

    Q. Yang, Q. Chen, W. Wang, B. Hu, and M. Zhang. Enhancing multi-modal multi-hop question answering via structured knowledge and unified retrieval-generation. InProceedings of the 31st ACM International Conference on Multimedia, pages 5223–5234, 2023

    2023

  34. [34]

    Chiang, Z

    W.-L. Chiang, Z. Li, Z. Lin, Y . Sheng, Z. Wu, H. Zhang, L. Zheng, S. Zhuang, Y . Zhuang, J. E. Gonzalez, I. Stoica, and E. P. Xing. Vicuna: An open-source chatbot impressing GPT-4 with 90%* ChatGPT quality. LMSYS Blog, 2023. Available athttps://lmsys.org/blog/2023-03-30-vicuna/. 11

    2023

  35. [35]

    Touvron, L

    H. Touvron, L. Martin, K. Stone, P. Albert, A. Almahairi, Y . Babaei, N. Bashlykov, S. Batra, P. Bhargava, S. Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models.arXiv preprint arXiv:2307.09288, 2023

    Pith/arXiv arXiv 2023

  36. [36]

    G. Wang, S. Cheng, X. Zhan, X. Li, S. Song, and Y . Liu. OpenChat: Advancing open-source language models with mixed-quality data. InInternational Conference on Learning Representations, 2024

    2024

  37. [37]

    Zhang, Y

    L. Zhang, Y . Wu, F. Mo, J.-Y . Nie, and A. Agrawal. MoqaGPT: Zero-shot multi-modal open-domain ques- tion answering with large language model. InFindings of the Association for Computational Linguistics: EMNLP, pages 1195–1210, 2023

    2023

  38. [38]

    P. Wang, A. Yang, R. Men, J. Lin, S. Bai, Z. Li, J. Ma, C. Zhou, J. Zhou, and H. Yang. OFA: Unifying architectures, tasks, and modalities through a simple sequence-to-sequence learning framework. In Proceedings of the 39th International Conference on Machine Learning, pages 23318–23340, 2022

    2022

  39. [39]

    Y . Hu, H. Hua, Z. Yang, W. Shi, N. A. Smith, and J. Luo. PromptCap: Prompt-guided image captioning for VQA with GPT-3. InProceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 2963–2975, 2023

    2023

  40. [40]

    Cheng, T

    Z. Cheng, T. Xie, P. Shi, C. Li, R. Nadkarni, Y . Hu, C. Xiong, D. Radev, M. Ostendorf, L. Zettlemoyer, N. A. Smith, and T. Yu. Binding language models in symbolic languages. InInternational Conference on Learning Representations, 2023

    2023

  41. [41]

    Loison, Q

    A. Loison, Q. Macé, A. Edy, V . Xing, T. Balough, G. Moreira, B. Liu, M. Faysse, C. Hudelot, and G. Viaud. ViDoRe V3: A comprehensive evaluation of retrieval augmented generation in complex real-world scenarios.arXiv preprint arXiv:2601.08620, 2026. 12 A Candidate-ranking and end-task breakdowns This section collects evaluation results onMultimodalQAfor t...

    Pith/arXiv arXiv 2026