Pith. sign in

REVIEW 4 major objections 4 minor 1 cited by

MRAMG-Bench: A Comprehensive Benchmark for Advancing Multimodal Retrieval-Augmented Multimodal Generation

T0 review · 4 major / 4 minor · reviewed 2026-08-08 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read MRAMG-Bench supplies 4,800 human-annotated QA pairs across six datasets, and the paper argues it is the first benchmark to require models to decide the number, selection, and ordering of images in an answer.

desk verdict A worthy benchmark for multimodal RAG generation, but the GPT-4o self-generation/self-judgment loop needs tightening before the LLM-based scores are treated as independent quality measures. read the letter →

arxiv 2502.04176 v2 pith:BW7N2FYE submitted 2025-02-06 cs.LG cs.IR

Qinhan Yu , Zhiyou Xiao , Binghui Li , Zhengren Wang , Chong Chen , Wentao Zhang This is my paper · ORCID
classification cs.LGcs.IR
keywords multimodalretrieval-augmentedgenerationanswerbenchmarkconstructionimage-textinterleavedanswersimageorderingevaluationmetricslargelanguagemodels
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper introduces the Multimodal Retrieval-Augmented Multimodal Generation (MRAMG) task, in which a model answers a text query with an answer that interleaves generated text and images taken from a retrieved multimodal corpus. To evaluate this task, it builds MRAMG-Bench: 4,800 human-annotated QA pairs over six datasets in the Web, Academia, and Lifestyle domains, with 4,346 documents and 14,190 images. The paper's central claim is that this is the first benchmark forcing a model to autonomously decide how many images to use, which images to select, and in what order to place them. It also contributes a three-part generation framework and a metric suite that grades image choice and ordering separately from text quality.

What carries the argument

The carrying object is the gold multimodal answer, defined as an interleaved sequence of text and images with a definite image count, selection, and order. Around this object the paper builds a multi-stage construction pipeline (data selection and preprocessing, QA generation and refinement, three-step human quality review) and a decomposition of evaluation into statistical metrics (image precision/recall/F1, ordering, BERTScore) plus LLM-based judgments. The generation framework contributes three strategies: an LLM method that substitutes image captions and surrounding context as placeholders, an MLLM method that feeds selected images directly, and a rule-based method that matches sentences to images via a weighted bipartite graph solved with Edmonds' blossom algorithm.

What would settle it

Independently re-annotate a random sample of the web and recipe QA pairs with human annotators who do not see the GPT-4o gold answers, then compare the image sets and orders they choose; if the alternative human answers shift model rankings by more than the reported gaps, the benchmark's gold standard is not stable enough for fine-grained comparison.

Watch

Extended reading notes

Core claim

The benchmark is built so that every question has a gold multimodal answer: a reference response containing both text blocks and designated images in a specified order. Using these gold answers, image precision, image recall, image F1, and an image-ordering score based on weighted edit distance measure whether the model picked the right images in the right sequence, while BERTScore and LLM-based judgments assess textual and overall quality. Across 11 generative models, the paper finds that overall performance drops as datasets become more image-dense and that no tested model reaches a passing image-ordering score on the lifestyle domain. It concludes that the MRAMG task is measurable, that open questions remain in image-order reasoning, and that a rule-based insertion framework is a cheap but weaker alternative to direct generation.

Load-bearing premise

The benchmark's validity rests on the assumption that GPT-4o-drafted gold answers for the web and recipe datasets, after only review by human annotators, are correct, representative, and unbiased examples of what a good multimodal answer should look like.

Editorial extensions

If this is right

  • Model rankings on MRAMG-Bench can be decomposed into retrieval quality, image-selection quality, image-order quality, and text quality, so failures can be attributed to a specific stage.
  • Because image ordering scores are low for all 11 tested models, ordered multi-image answers should be treated as an unsolved subproblem rather than a solved generation capability.
  • The LLM-based method, which gives the model image captions and surrounding context as text, is competitive with or better than directly feeding images to MLLMs on most current models, indicating that contextual text carries much of the image-selection signal.
  • The rule-based insertion method is considerably cheaper and more stable, but its weaker performance on image-sparse web data suggests it is best used in resource-constrained settings.
  • Open-source models at roughly 70B parameters approach closed-source models on easy web data, but the gap widens sharply on academic and lifestyle data.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If GPT-4o's image-order preferences are not the only reasonable ones, valid alternative orderings could be counted as errors; a human-agreement study on a sample of gold answers would put an upper bound on the reliability of the ordering metric.
  • The benchmark's structure could be reused to study whether retrieval quality is the bottleneck: on manual-style documents visual recall is lowest, so improving retrieval may be a direct lever for improving generation scores.
  • A testable extension is to add an explicit ordering-aware training or decoding objective; the paper's low order scores predict such an objective would yield the largest gains.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The paper introduces the MRAMG task, in which a model must produce interleaved text-and-image answers from retrieved multimodal documents, and presents MRAMG-Bench, a benchmark with 4,800 QA pairs across six datasets in the Web, Academia, and Lifestyle domains. The construction pipeline combines GPT-4o-based generation with human review, and the paper proposes both statistical and LLM-based evaluation metrics, a generation framework with LLM-based, MLLM-based, and rule-based variants, and results for 11 generative models. The central claims are that MRAMG-Bench is the first benchmark for this task with human-annotated gold answers, hierarchical difficulty levels, and order-based reasoning, and that its evaluation framework is statistically grounded and rigorous.

Significance. If the validity concerns are resolved, MRAMG-Bench would be a valuable community resource: it addresses a real gap, provides a relatively large set of QA pairs with interleaved text and images, and ships an evaluation harness and strong baselines. The paper is careful to separate statistical metrics from LLM-based judgments, and the manual construction of the Academic and Manual subsets is a genuine strength. The explicit difficulty levels, the inclusion of text-only QA pairs to test image suppression, and the comparison of three generation strategies are thoughtful design choices. The main risk is the circularity between GPT-4o-generated gold answers and GPT-4o-based judging, which must be quantified before the rankings in Table 4 can be interpreted as measuring general answer quality rather than agreement with GPT-4o.

major comments (4)
  1. [§4.2.2, §6.1, Table 4] The reference answers for the three Web datasets (MRAMG-Wit, MRAMG-Wiki, MRAMG-Web; 1,850 QA pairs) and for MRAMG-Recipe step-specific questions are generated by GPT-4o, while the four LLM-based metrics (Image Relevance, Image Effectiveness, Image Position Score, Comprehensive Score) are scored by GPT-4o as judge. Because the same model family constructs the gold answers and evaluates model outputs, the Average scores in Table 4 may reward agreement with GPT-4o's stylistic preferences for image selection and placement rather than independently verifiable answer quality. Please quantify the human review process (e.g., number of corrected or discarded answers), report agreement between GPT-4o judgments and human judgments on a random sample, and preferably re-run the main comparisons with an alternative judge model or with human-written gold answers for a held-out subset.
  2. [§6.2.2] The four LLM-based metrics are only named, with no scoring rubrics, prompt templates, or aggregation formulas; the pointer to the GitHub repository is not sufficient for a self-contained benchmark paper. Since these metrics contribute to the reported Average scores in Table 4, the evaluation is not reproducible as written. Please include the exact prompts, scoring scales, and any post-processing steps in the paper or an appendix.
  3. [§5.2.3, §6.1] The rule-based generation method depends on thresholds alpha* and beta*, and on the weight lambda in the edge construction and Equation (2), but no values for these hyperparameters are reported. Likewise, the experimental setting fixes top-k=10 and chunk size 256 but does not report the top-N value used for MLLM input in Section 5.2.2. Without these values and a sensitivity analysis, the rule-based results in Table 4 cannot be reproduced and the method comparisons are conditional on unreported choices.
  4. [§4.3] The quality-check section describes a three-step human review but provides no inter-annotator agreement, no number of annotators per dataset, and no quantitative outcome such as the fraction of QA pairs that required correction. Given that the abstract calls the benchmark "human-annotated" while large parts of the gold answers are GPT-4o-generated and only reviewed by humans, this evidence is needed to support the central validity claim.
minor comments (4)
  1. [§6.3.2] The statement that no model achieves a "passing score" on the ordering metric presupposes a passing threshold that is never defined; please either define it or remove the qualitative claim.
  2. [Table 2] The definition of "Multimodal Element Density" is not stated; please specify the exact formula (e.g., images per token or per character) so the column can be interpreted.
  3. [§6.2.1] Context Recall is said to use LLMs, but the specific judge model and prompt are not specified; please state them explicitly for reproducibility.
  4. [§5.2.3] The text contains typographical errors such as "egde" and "infomation"; additionally, the phrase "each egde" in Step 2 should read "each edge."

Circularity Check

1 steps flagged · score 6.0 of 10

Benchmark gold answers and LLM-based judge are both GPT-4o; for Web QA and Recipe step answers the human pass is review-only, so part of the evaluation loop is self-referential.

  1. other [Section 4.2.2 (Answer Generation), Section 4.3 (Data Quality Check), Section 6.1 (Experimental Baselines and Settings), Section 6.2.2 (Generation Evaluation)]
    "For Web Data, given the generated or original questions Q and corresponding context C, we generate image-text interleaved answers A using GPT-4o following a CoT reasoning strategy [43] ... use answers generated by GPT-4o as the answers of step-specific questions ... a group of annotators review all QA pairs ... we also integrate GPT-4o for QA Evaluation ... In the evaluation stage, we utilize GPT-4o [29] as the judging model to assess the performance of LLM-based metrics."

    The gold answers for the three Web datasets (1,850 QA pairs) and for MRAMG-Recipe step-specific questions are produced by GPT-4o, and GPT-4o is also used in the QA-quality gate and as the judge for the four LLM-based metrics (Image Relevance, Image Effectiveness, Image Position Score, Comprehensive Score). Human annotators review and correct issues but do not rebuild these answers from scratch, so the loop is not fully broken for those subsets.

full rationale

MRAMG-Bench is a new benchmark rather than a derivation with fitted parameters, and there is no load-bearing self-citation chain: references [42] and [49] include present authors but only as general RAG background. The central circularity issue is that a large slice of the gold standard and the LLM-based judge come from the same model family. Section 4.2.2 generates Web and Recipe step-specific gold answers with GPT-4o, Section 4.3 adds GPT-4o to the QA-quality check, and Section 6.1 uses GPT-4o as the judge for the four LLM-based metrics. For those subsets and metrics, a high score means 'resembles GPT-4o's answer and is preferred by GPT-4o as judge', not an independent external criterion. This is mitigated by human review/correction of all QA pairs, human-formulated gold answers for MRAMG-Arxiv and MRAMG-Manual, and objective statistical metrics (image precision, recall, ordering score, ROUGE-L, BERTScore) that do not use GPT-4o as judge. These anchors make the circularity partial, so a mid-range score of 6 is appropriate. The task formulation, generation framework, and novelty claims are not themselves circular; only the evaluation loop for the LLM-based metrics is partly self-referential.

Assumptions & free parameters 6 free parameters · 5 assumptions · 0 invented entities

The central resource rests on two main pillars: the benchmark data and the evaluation metrics. The data pillar depends on GPT-4o-generated content plus human review, on the trustworthiness of the source corpora (Wit, WikiWeb2M, WebQA, arXiv, RecipeQA, manuals), and on the assumption that each question has a single correct image set and order. The metric pillar depends on BERTScore, Rouge-L, weighted edit distance, and on GPT-4o's judgments for the LLM-based metrics. The rule-based generation method adds three unreported hyperparameters. No new entities are introduced.

free parameters (6)
  • alpha* (string similarity threshold)
    Used in Section 5.2.3 to filter edges in the bipartite graph for image insertion; value not reported, presumably tuned on development data.
  • beta* (semantic similarity threshold)
    Same role as alpha* but for BGE-M3 semantic similarity; value not reported.
  • lambda (BLEU vs semantic weight)
    Weights the edge score in w_{k,l} = lambda*alpha + (1-lambda)*beta; no value or sensitivity analysis reported.
  • top_k retrieved documents/chunks = 10
    Retrieval uses top-k=10 chunks in Section 6.1; no ablations over k.
  • top-N images for MLLM input
    Section 5.2.2 selects top-N images by CLIP similarity but N is not specified.
  • chunk size = 256
    Documents chunked to 256 tokens with SentenceSplitter; choice not justified or varied.
assumptions (5)
  • domain assumption BGE-M3 embedding cosine similarity reliably ranks multimodal documents relevant to a query
    Invoked in Section 5.1 to build D*_q; no gold retrieval baseline or human relevance judgments are reported.
  • domain assumption GPT-4o can generate accurate, context-grounded reference answers and valid QA pairs
    Section 4.2 uses GPT-4o with a CoT protocol to create a large fraction of the benchmark's ground truth; if GPT-4o hallucinates or inserts images incorrectly, the gold standard inherits these errors.
  • domain assumption Human annotators correctly identify and correct GPT-4o errors
    Section 4.3 relies on human review but reports no inter-annotator agreement, annotator counts per dataset, or error-rate statistics.
  • domain assumption The ground-truth image set and image order are the unique correct answer
    Metrics in Section 6.2 compare only against ground-truth image lists and the ground-truth order, penalizing alternative but valid images or orders.
  • domain assumption The corpus contains all information needed to answer the questions
    QA pairs are generated from the collected documents, so questions are answerable by construction, which may not reflect open-domain user queries.

how reviews work

0 comments
Cite this review

Pith. "Pith review of MRAMG-Bench: A Comprehensive Benchmark for Advancing Multimodal Retrieval-Augmented Multimodal Generation." pith.science (2026). https://pith.science/paper/BW7N2FYE

@misc{pith2026250204176,
  author       = {Pith},
  title        = {Pith review of: MRAMG-Bench: A Comprehensive Benchmark for Advancing Multimodal Retrieval-Augmented Multimodal Generation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/BW7N2FYE}},
  note         = {Machine review of arXiv:2502.04176}
}
read the original abstract

Recent advances in Retrieval-Augmented Generation (RAG) have significantly improved response accuracy and relevance by incorporating external knowledge into Large Language Models (LLMs). However, existing RAG methods primarily focus on generating text-only answers, even in Multimodal Retrieval-Augmented Generation (MRAG) scenarios, where multimodal elements are retrieved to assist in generating text answers. To address this, we introduce the Multimodal Retrieval-Augmented Multimodal Generation (MRAMG) task, in which we aim to generate multimodal answers that combine both text and images, fully leveraging the multimodal data within a corpus. Despite growing attention to this challenging task, a notable lack of a comprehensive benchmark persists for effectively evaluating its performance. To bridge this gap, we provide MRAMG-Bench, a meticulously curated, human-annotated benchmark comprising 4,346 documents, 14,190 images, and 4,800 QA pairs, distributed across six distinct datasets and spanning three domains: Web, Academia, and Lifestyle. The datasets incorporate diverse difficulty levels and complex multi-image scenarios, providing a robust foundation for evaluating the MRAMG task. To facilitate rigorous evaluation, MRAMG-Bench incorporates a comprehensive suite of both statistical and LLM-based metrics, enabling a thorough analysis of the performance of generative models in the MRAMG task. Additionally, we propose an efficient and flexible multimodal answer generation framework that can leverage LLMs/MLLMs to generate multimodal responses. Our datasets and complete evaluation results for 11 popular generative models are available at https://github.com/MRAMG-Bench/MRAMG.

Figures

Figures reproduced from arXiv: 2502.04176 by the authors.

Figure 1
Figure 1. Illustration of the MRAMG task (above), with scenarios below showing how integrating text and images enhances [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. The MRAMG-Bench construction pipeline consists of three stages: (1) Data Selection and Preprocessing, where data [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Comparison of the generative performance of two [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Reasoning RAG via System 1 or System 2: A Survey on Reasoning Agentic Retrieval-Augmented Generation for Industry Challenges

    cs.AI 2025-06 unverdicted novelty 3.0 of 10

    A review that classifies Reasoning Agentic RAG into predefined (System 1-like) and agentic (System 2-like) workflows, surveying their designs and training strategies.

Reference graph

Works this paper leans on

52 extracted references · 14 canonical work pages · cited by 1 Pith paper

  1. [1]

    Anthropic. 2024. Claude 3.5 Sonnet. https://www.anthropic.com/news/claude-3- 5-sonnet

    2024

  2. [2]

    Zechen Bai, Pichao Wang, Tianjun Xiao, Tong He, Zongbo Han, Zheng Zhang, and Mike Zheng Shou. 2024. Hallucination of multimodal large language models: A survey. arXiv preprint arXiv:2404.18930 (2024)

    arXiv 2024

  3. [3]

    Andrea Burns, Krishna Srinivasan, Joshua Ainslie, Geoff Brown, Bryan A Plum- mer, Kate Saenko, Jianmo Ni, and Mandy Guo. 2023. Wikiweb2m: A page-level multimodal wikipedia dataset. arXiv preprint arXiv:2305.05432 (2023)

    work page Pith review arXiv 2023

  4. [4]

    Yingshan Chang and Yonatan Bisk. 2022. WebQA: A Multimodal Multihop NeurIPS Challenge. In NeurIPS 2021 Competitions and Demonstrations Track . PMLR, 232–245

    work page 2022

  5. [5]

    Jianlv Chen, Shitao Xiao, Peitian Zhang, Kun Luo, Defu Lian, and Zheng Liu. 2024. Bge m3-embedding: Multi-lingual, multi-functionality, multi-granularity text embeddings through self-knowledge distillation. arXiv preprint arXiv:2402.03216 (2024)

    arXiv 2024

  6. [6]

    Wenhu Chen, Hexiang Hu, Xi Chen, Pat Verga, and William W Cohen. 2022. Murag: Multimodal retrieval-augmented generator for open question answering over images and text. arXiv preprint arXiv:2210.02928 (2022)

    arXiv 2022

  7. [7]

    Zhe Chen, Weiyun Wang, Yue Cao, Yangzhou Liu, Zhangwei Gao, Erfei Cui, Jinguo Zhu, Shenglong Ye, Hao Tian, Zhaoyang Liu, et al . 2024. Expanding performance boundaries of open-source multimodal models with model, data, and test-time scaling. arXiv preprint arXiv:2412.05271 (2024)

    arXiv 2024

  8. [8]

    Jacob Devlin. 2018. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805 (2018)

    arXiv 2018

Show all 52 references
  1. [9]

    Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. 2024. The llama 3 herd of models. arXiv preprint arXiv:2407.21783 (2024)

    2024 arXiv

  2. [10]

    Jack Edmonds. 1965. Paths, Trees, and Flowers. Canadian Journal of Mathematics 17 (1965), 449–467. doi:10.4153/CJM-1965-045-4

    1965 doi

  3. [11]

    Shahul Es, Jithin James, Luis Espinosa-Anke, and Steven Schockaert. 2023. Ra- gas: Automated evaluation of retrieval augmented generation. arXiv preprint arXiv:2309.15217 (2023)

    2023 arXiv

  4. [12]

    Liangke Gui, Borui Wang, Qiuyuan Huang, Alex Hauptmann, Yonatan Bisk, and Jianfeng Gao. 2021. Kat: A knowledge augmented transformer for vision-and- language. arXiv preprint arXiv:2112.08614 (2021)

    2021 arXiv

  5. [13]

    Aman Gupta, Anup Shirgaonkar, Angels de Luis Balaguer, Bruno Silva, Daniel Holstein, Dawei Li, Jennifer Marsman, Leonardo O Nunes, Mahsa Rouzbahman, Morris Sharp, et al. 2024. RAG vs Fine-tuning: Pipelines, Tradeoffs, and a Case Study on Agriculture. arXiv preprint arXiv:2401....

    2024 arXiv

  6. [14]

    Lei Huang, Weijiang Yu, Weitao Ma, Weihong Zhong, Zhangyin Feng, Haotian Wang, Qianglong Chen, Weihua Peng, Xiaocheng Feng, Bing Qin, et al. 2023. A survey on hallucination in large language models: Principles, taxonomy, chal- lenges, and open questions. arXiv preprint arXiv:2...

    2023 arXiv

  7. [15]

    Dongzhi Jiang, Renrui Zhang, Ziyu Guo, Yanmin Wu, Jiayi Lei, Pengshuo Qiu, Pan Lu, Zehui Chen, Guanglu Song, Peng Gao, et al. 2024. Mmsearch: Benchmarking the potential of large models as multi-modal search engines. arXiv preprint arXiv:2409.12959 (2024)

    2024 arXiv

  8. [16]

    Karen Sparck Jones. 1973. Index term weighting. Information storage and retrieval 9, 11 (1973), 619–633

    1973

  9. [17]

    Mandar Joshi, Eunsol Choi, Daniel Weld, and Luke Zettlemoyer. 2017. TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehension. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) . As...

    2017

  10. [18]

    Andreas Koukounas, Georgios Mastrapas, Bo Wang, Mohammad Kalim Akram, Sedigheh Eslami, Michael Günther, Isabelle Mohr, Saba Sturua, Scott Martens, Nan Wang, et al. 2024. jina-clip-v2: Multilingual Multimodal Embeddings for Text and Images. arXiv preprint arXiv:2412.08802 (2024)

    2024 arXiv

  11. [19]

    Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, et al. 2019. Natural questions: a benchmark for question answering research. Transactions of the Association for C...

    2019

  12. [20]

    Patrick S. H. Lewis, Ethan Perez, Aleksandra Piktus, et al . 2020. Retrieval- Augmented Generation for Knowledge-Intensive NLP Tasks. In NeurIPS

    2020

  13. [21]

    Chin-Yew Lin. 2004. ROUGE: A Package for Automatic Evaluation of Summaries. In Text Summarization Branches Out. Association for Computational Linguistics, Barcelona, Spain, 74–81. https://aclanthology.org/W04-1013/

    2004

  14. [22]

    Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Cheng- gang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, et al. 2024. DeepSeek-V3 Technical Report. arXiv preprint arXiv:2412.19437 (2024)

    2024 arXiv

  15. [23]

    Haotian Liu, Kilho Son, Jianwei Yang, Ce Liu, Jianfeng Gao, Yong Jae Lee, and Chunyuan Li. 2023. Learning customized visual models with retrieval-augmented knowledge. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 15148–15158

    2023

  16. [24]

    Jerry Liu. 2022. LlamaIndex. doi:10.5281/zenodo.1234

    2022 doi

  17. [25]

    Zi-Ao Ma, Tian Lan, Rong-Cheng Tu, Yong Hu, Heyan Huang, and Xian-Ling Mao

  18. [26]

    Kenneth Marino, Mohammad Rastegari, Ali Farhadi, and Roozbeh Mottaghi. 2019. Ok-vqa: A visual question answering benchmark requiring external knowledge. In Proceedings of the IEEE/cvf conference on computer vision and pattern recognition. 3195–3204

    2019

  19. [27]

    Tri Nguyen, Mir Rosenberg, Xia Song, Jianfeng Gao, Saurabh Tiwary, Rangan Majumder, and Li Deng. 2016. MS MARCO: A Human Generated MAchine Reading COmprehension Dataset. choice 2640 (2016), 660

    2016

  20. [28]

    OpenAI. 2024. GPT-4o mini: advancing cost-efficient intelligence. OpenAI Blog (2024). https://openai.com/index/gpt-4o-mini-advancing-cost-efficient- intelligence//

    2024

  21. [29]

    OpenAI. 2024. Hello GPT-4o.OpenAI Blog (2024). https://openai.com/index/hello- gpt-4o/

    2024

  22. [30]

    Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeting of the Association for Computational Linguistics . 311–318

    2002

  23. [31]

    Fabio Petroni, Aleksandra Piktus, Angela Fan, Patrick Lewis, Majid Yazdani, Nicola De Cao, James Thorne, Yacine Jernite, Vladimir Karpukhin, Jean Maillard, et al. 2020. KILT: a benchmark for knowledge intensive language tasks. arXiv preprint arXiv:2009.02252 (2020)

    2020 arXiv

  24. [32]

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. 2021. Learning transferable visual models from natural language supervision. In International conference on machine learni...

    2021

  25. [33]

    Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016. SQuAD: 100,000+ Questions for Machine Comprehension of Text. In Proceed- ings of the 2016 Conference on Empirical Methods in Natural Language Processing . Association for Computational Linguistics

    2016

  26. [34]

    Dustin Schwenk, Apoorv Khandelwal, Christopher Clark, Kenneth Marino, and Roozbeh Mottaghi. 2022. A-okvqa: A benchmark for visual question answering using world knowledge. In European conference on computer vision . Springer, 146–162

    2022

  27. [35]

    Anshumali Shrivastava and Ping Li. 2014. In defense of minhash over simhash. In Artificial Intelligence and Statistics. PMLR, 886–894

    2014

  28. [36]

    Krishna Srinivasan, Karthik Raman, Jiecao Chen, Michael Bendersky, and Marc Najork. 2021. Wit: Wikipedia-based image text dataset for multimodal multi- lingual machine learning. In Proceedings of the 44th international ACM SIGIR conference on research and development in inform...

    2021

  29. [37]

    Alon Talmor, Ori Yoran, Amnon Catav, Dan Lahav, Yizhong Wang, Akari Asai, Gabriel Ilharco, Hannaneh Hajishirzi, and Jonathan Berant. 2021. Multimodalqa: Complex question answering over text, tables and images. arXiv preprint arXiv:2104.06039 (2021)

    2021 arXiv

  30. [38]

    Gemini Team, Petko Georgiev, Ving Ian Lei, Ryan Burnell, Libin Bai, Anmol Gulati, Garrett Tanzer, Damien Vincent, Zhufeng Pan, Shibo Wang, et al. 2024. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context. arXiv preprint arXiv:2403.05530 (2024)

    2024 arXiv

  31. [39]

    Bin Wang, Chao Xu, Xiaomeng Zhao, Linke Ouyang, Fan Wu, Zhiyuan Zhao, Rui Xu, Kaiwen Liu, Yuan Qu, Fukai Shang, et al. 2024. Mineru: An open-source solution for precise document content extraction. arXiv preprint arXiv:2409.18839 (2024)

    2024 arXiv

  32. [40]

    Jiaqi Wang, Hanqi Jiang, Yiheng Liu, Chong Ma, Xu Zhang, Yi Pan, Mengyuan Liu, Peiran Gu, Sichen Xia, Wenjun Li, et al. 2024. A comprehensive review of multimodal large language models: Performance and challenges across different tasks. arXiv preprint arXiv:2408.01319 (2024)

    2024 arXiv

  33. [41]

    Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, et al. 2024. Qwen2-vl: Enhancing vision-language model’s perception of the world at any resolution.arXiv preprint arXiv:2409.12191 (2024)

    2024 arXiv

  34. [42]

    Zhengren Wang, Qinhan Yu, Shida Wei, Zhiyu Li, Feiyu Xiong, Xiaoxing Wang, Simin Niu, Hao Liang, and Wentao Zhang. 2024. QAEncoder: Towards Aligned Representation Learning in Question Answering System. arXiv preprint arXiv:2409.20434 (2024)

    2024 arXiv

  35. [43]

    Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. 2022. Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems 35 (2022), 24824–24837

    2022

  36. [44]

    Shitao Xiao, Zheng Liu, Peitian Zhang, Niklas Muennighoff, Defu Lian, and Jian-Yun Nie. 2024. C-Pack: Packaged Resources To Advance General Chinese Embedding. arXiv:2309.07597 [cs.CL] https://arxiv.org/abs/2309.07597

    2024 arXiv

  37. [45]

    Semih Yagcioglu, Aykut Erdem, Erkut Erdem, and Nazli Ikizler-Cinbis. 2018. Recipeqa: A challenge dataset for multimodal comprehension of cooking recipes. arXiv preprint arXiv:1809.00812 (2018)

    2018 arXiv

  38. [46]

    Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William Cohen, Ruslan Salakhutdinov, and Christopher D Manning. 2018. HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answering. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language ...

    2018

  39. [47]

    Pan Zhang, Xiaoyi Dong, Bin Wang, Yuhang Cao, Chao Xu, Linke Ouyang, Zhiyuan Zhao, Haodong Duan, Songyang Zhang, Shuangrui Ding, et al. 2023. MRAMG-Bench: A Comprehensive Benchmark for Advancing Multimodal Retrieval-Augmented Multimodal Generation SIGIR ’25, July 13–18, 2025, ...

    2023 arXiv

  40. [48]

    Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q Weinberger, and Yoav Artzi. 2019. Bertscore: Evaluating text generation with bert. arXiv preprint arXiv:1904.09675 (2019)

    2019 arXiv

  41. [49]

    Penghao Zhao, Hailin Zhang, Qinhan Yu, Zhengren Wang, Yunteng Geng, Fangcheng Fu, Ling Yang, Wentao Zhang, and Bin Cui. 2024. Retrieval-augmented generation for ai-generated content: A survey. arXiv preprint arXiv:2402.19473 (2024)

    2024 arXiv

  42. [50]

    Kunlun Zhu, Yifan Luo, Dingling Xu, Ruobing Wang, Shi Yu, Shuo Wang, Yukun Yan, Zhenghao Liu, Xu Han, Zhiyuan Liu, et al. 2024. Rageval: Scenario specific rag evaluation dataset generation framework. arXiv preprint arXiv:2408.01262 (2024)

    2024 arXiv

  43. [51]

    Zhengyuan Zhu, Daniel Lee, Hong Zhang, Sai Sree Harsha, Loic Feujio, Akash Maharaj, and Yunyao Li. 2024. Murar: A simple and effective multimodal retrieval and answer refinement framework for multimodal question answering. arXiv preprint arXiv:2408.08521 (2024)

    2024 arXiv

  44. [2024]

    arXiv preprint arXiv:2411.16365 (2024)

    Multi-modal Retrieval Augmented Multi-modal Generation: A Benchmark, Evaluate Metrics and Strong Baselines. arXiv preprint arXiv:2411.16365 (2024)

    2024 arXiv

Pith tools

Reviewed August 8, 2026 · model on record in the stance chip above.