Pith. sign in

REVIEW 4 major objections 5 minor 58 references

An agent trained with reinforcement learning can decide when to search and where to insert authentic images, beating static retrieve-then-generate pipelines on verified interleaved answers.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · grok-4.5

2026-07-31 19:13 UTC pith:NDLH4S2S

load-bearing objection Solid agentic-RL recipe for verified image insert on MRAMG-Bench; real gains over static RAG and matched SFT, but SOTA talk still sits on one suite and a GPT-4o C.S. judge. the 4 major comments →

arxiv 2607.28055 v1 pith:NDLH4S2S submitted 2026-07-30 cs.IR

VIG-RL: Learning to Search and Insert for Verified Image Grounding

Qinhan Yu , Jun Guang , Chong Chen , Wentao Zhang This is my paper
classification cs.IR
keywords verified image groundinginterleaved multimodal generationretrieval-augmented generationagentic reinforcement learningReActGRPOimage insertionmultimodal agents
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Knowledge-heavy questions often need real photographs or diagrams woven into the text, not model-drawn pictures that can invent wrong details. Static retrieval systems fetch evidence once and then write, so they cannot decide mid-answer whether more search is needed or exactly where a verified image belongs. This paper treats that whole search–select–insert loop as a decision process an agent runs inside a think-then-act cycle, and trains the agent with reinforcement learning under a reward that checks format, text quality, image-selection precision, and whether the right images were even retrieved. On a six-part benchmark with ground-truth placements, the trained agent outperforms the same backbone under fixed retrieval, strong commercial models in the same static setup, and a matched imitation baseline. The practical claim is that reliable multimodal answers come from learning the policy of when and where to ground, not from better one-shot retrieval alone.

Core claim

Verified Image Grounding can be solved by training a multimodal agent, inside a ReAct-style loop with text-search, image-search, and answer actions, via group-relative policy optimization under a gated composite reward that multiplies a hard format check by text-judge quality, rule-based image-insertion precision, and search recall; the resulting policy sets a new state of the art on MRAMG-Bench, lifting image selection and overall interleaved quality well above static RAG and matched supervised trajectories.

What carries the argument

The gated composite reward R = r_format · (r_txt,LLM + r_img,pre + r_search) that forces valid tool syntax before scoring text quality, precision of inserted image IDs against gold, and recall of gold images in the retrieved pool, together with symbolic <img:id> tags that let the model place only retrieved assets.

Load-bearing premise

That large gains on one public suite with explicit placement labels, about a thousand training questions, and a model-based comprehensive-score judge are enough to establish a general method for verified image grounding.

What would settle it

Train and evaluate the same agent on a different verified-image benchmark that uses independent placement annotations and a non-overlapping judge; if the advantage over tuned static RAG and matched SFT on image F1 and placement-sensitive scores disappears, the central claim fails.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Interleaved answer systems can replace fixed retrieve-then-generate pipelines with a learned policy that chooses when to call text or image search.
  • Placement and ordering of images can improve without any position labels if retrieval and selection are first made reliable under RL.
  • Smaller backbones (around 4B) can reach competitive verified grounding once the search–insert policy is trained, not only large models.
  • Context-anchored image retrieval plus optional text search becomes a practical multi-hop evidence path for diagrams whose meaning lives in captions.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The same ReAct-plus-gated-reward pattern could transfer to other provenance-constrained outputs, such as citing only retrieved tables or code snippets inside long answers.
  • Because the image reward is pure precision on IDs, the method may under-insert on open-web corpora where gold sets are incomplete; coverage-aware terminal rewards would be a natural next stress test.
  • If process recall and terminal precision stay decoupled, agents might learn shorter, higher-precision trajectories that cut tool cost in production assistants.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper introduces Verified Image Grounding (VIG)—interleaved text–image generation that may insert only externally retrieved authentic images—and proposes VIG-RL, a ReAct-style agent with actions for text search, image search, and answer generation. The agent is trained with GRPO under a gated composite reward R = r_format · (r_txt,LLM + r_img,pre + r_search), combining format validity, a text-only LLM judge, rule-based image-insertion precision against gold ID sets, and search-process recall of gold images. Symbolic <img:id> tags enforce provenance. On MRAMG-Bench (six splits; 1.1k Web/Wiki/Arxiv training instances), VIG-RL-8B reports large gains over same-backbone static RAG (macro Image F1 57.7→78.9, C.S. 56.4→78.1), zero-shot agentic MLLMs, a matched best-of-five Gemini SFT baseline, and several strong commercial RAG systems, with supporting reward and retrieval ablations (Table 2) and claimed emergent order gains without positional rewards.

Significance. If the results hold under broader evaluation, the work is a clear step beyond static retrieve-then-generate VIG pipelines: it jointly learns when to search, what to select, and where to insert verified visuals, with a carefully factorized hybrid reward that avoids the EM sparsity and pure-judge hacking modes shown in the ablations. Strengths include the matched SFT control on the same 1.1k instances and tools, rule-based image precision/search recall that anchor multimodal claims outside pure self-judgment, context-anchored retrieval vs CLIP-style matching, and consistent 4B/8B gains including OOD Manual/Recipe Order. The problem framing (authentic evidence vs generative hallucination) is practically important for knowledge-intensive multimodal assistants.

major comments (4)
  1. [Experiments; Table 1; Method (Image Insertion Reward); Metrics] The SOTA claim for VIG is supported almost entirely by MRAMG-Bench (Experiments; Table 1). Image F1 is set overlap with the same gold image-ID inventory used by r_img,pre in training (Method: Image Insertion Reward; Metrics). With only 1.1k curated in-domain instances and 720 GRPO steps, macro F1 gains partly measure success at retrieving and selecting that closed ID set after learning to trigger search, not independent insertion skill. OOD Wit/Manual/Recipe remain inside the same suite and annotation style. Establishing “new state-of-the-art for VIG” requires at least one external corpus/protocol or a held-out image inventory not seen as gold IDs at train time; otherwise the claim should be scoped to MRAMG-Bench.
  2. [Metrics; Further Analysis (Strict Reward Decoupling); Appendix Table 8] Placement, text–image alignment, and coherence—the quantities that separate “insert” from “retrieve-and-select”—are carried primarily by Comprehensive Score via GPT-4o (Metrics; Appendix Table 8). No human correlation, inter-annotator agreement, judge-model ablation, or sensitivity analysis is reported. Order Score on Manual/Recipe remains modest in absolute terms (32.2 / 40.9) even for VIG-RL-8B. Because the training reward is deliberately invariant to placement and order (Further Analysis), the central “search–selection–insertion policy” claim leans on an unvalidated LMM judge. A small human study or multi-judge agreement on C.S. (and on emergent Order) is load-bearing for the insertion half of the contribution.
  3. [Experimental Settings; Figure 4; Main Results (VIG-RL Surpasses Static RAG)] Static RAG depth is tuned (K∈{2,5,10}, default K=5; Figure 4) while the agent may issue multiple top-5 calls up to T_max=5. The paper argues this compares adaptive vs fixed-depth workflows rather than raw candidate counts, which is reasonable, but Table 1 still mixes systems that see very different evidence budgets and interaction compute. A budget-matched comparison (e.g., static RAG given the union of all passages/images the agent retrieved, or equal total retrieval calls) would clarify how much of the +21 C.S. gain is policy quality versus iterative evidence access. Without it, the gap vs “strongest static RAG oracle” (+5.7 C.S.) is the more credible headline and should be foregrounded.
  4. [Training via GRPO; SFT Baseline; Table 2; Figure 3] Training-time text reward uses Qwen3-8B-Instruct as judge; SFT demonstrations are best-of-five Gemini-2.5-Flash trajectories selected by Image F1 (Implementation Details). Eval C.S. uses GPT-4o. The ablations correctly show pure MLLM rewards collapse Image F1 via hacking (Table 2; Figure 6), and hybrid design is a genuine contribution—but dependence on proprietary/strong teacher rollouts for the SFT control and on a single family of LMM judges for open-ended text quality should be stress-tested (e.g., alternate text judge, human spot-check of r_txt,LLM). Otherwise residual teacher/judge bias remains a confounder for the RL-vs-SFT narrative in Figure 3.
minor comments (5)
  1. [Abstract; Introduction] Abstract and Introduction claim SOTA over “existing static baselines” without immediately noting that agentic zero-shot commercial models underperform static RAG of the same models (Table 1)—a useful finding that should be stated up front to avoid over-reading the agentic framing alone.
  2. [Table 1] M2IO-R1-3B is only partially comparable (missing metrics marked “-”). Either complete the comparison under a common protocol or move it to a limited side note so Table 1 is not visually uneven.
  3. [Gated Composite Reward; Table 2] Composite reward sums r_txt,LLM + r_img,pre + r_search with equal implicit weights after the format gate; sensitivity to relative weighting is not reported. A one-paragraph weight sweep would strengthen the “hybrid design” claim.
  4. [Related Work; Implementation Details] Figure 1 caption and related-work citations are helpful; ensure all arXiv-only concurrent works (e.g., MMSearch-R1, M2IO-R1) are clearly marked as concurrent and that BGE-M3 / veRL versions are pinned for reproducibility.
  5. [Throughout] Minor prose issues: spacing artifacts in the compiled text (“VIG-RLisoptimized”, “whenexternal”, etc.) and inconsistent capitalization of dataset names (Web/Wiki/Arxiv vs Wit). Clean for camera-ready.

Circularity Check

0 steps flagged

No load-bearing circular derivation: VIG-RL’s SOTA claim is empirical RL on held-out/OOD splits, not a prediction forced by construction from its inputs.

full rationale

This is an empirical agentic-RL systems paper, not a first-principles derivation. The policy is optimized with GRPO under a composite reward R = r_format · (r_txt,LLM + r_img,pre + r_search), then evaluated on held-out in-domain splits and three OOD datasets (Wit/Manual/Recipe) with no train–test overlap in required chunks or images. Matching a rule-based image-precision/F1 signal to gold ID sets at train and test time is standard supervised/RL practice on a labeled benchmark; it does not make test Image F1 equal the training input by construction, because the agent must still decide when to search, which queries to issue, and which retrieved IDs to insert on unseen questions. Text quality and Comprehensive Score use external LLM judges (Qwen3-8B train-time; GPT-4o eval), and a matched SFT baseline on best-of-five Gemini trajectories is reported as a control—none of which collapses the central claim into a tautology. Overlap of authors with MRAMG-Bench / M2IO-R1 is ordinary self-citation of prior benchmark and baseline work and is not load-bearing uniqueness or an ansatz smuggled in as external fact. Concerns about single-suite generalization and unvalidated GPT-4o C.S. are evaluation-validity issues, not circular reduction of a claimed derivation to its inputs. No self-definitional identity, fitted-constant-as-prediction, or uniqueness-from-authors step is present.

Axiom & Free-Parameter Ledger

4 free parameters · 5 axioms · 3 invented entities

The central claim rests on standard RL/MLLM tooling plus domain choices: ReAct+GRPO, a specific four-part gated reward, context-anchored retrieval with BGE-M3, MRAMG-Bench gold interleaved answers as truth, and LLM-as-judge text/C.S. signals. No new physical entities; free knobs are training/retrieval hyperparameters and reward factorization choices rather than fitted scientific constants.

free parameters (4)
  • Static RAG depth K (default 5) = K=5
    Chosen by tuning K∈{2,5,10} on held settings; used as the main static baseline operating point.
  • GRPO training hyperparameters = lr=2e-6, 20 epochs/720 steps, rollout.n=8, Tmax=5, KL=0
    Learning rate, epochs/steps, rollout.n, Tmax, KL coefficient set by authors for the reported runs.
  • Top-K per search call = Ktxt/Kimg=5
    Each agentic text/image call returns top-5 candidates; interacts with adaptive re-search policy.
  • Composite reward factorization weights = R=r_format·(r_txt+r_img,pre+r_search)
    Equal sum of r_txt,LLM + r_img,pre + r_search inside a multiplicative format gate; structure chosen ad hoc then ablated.
axioms (5)
  • domain assumption Verified images must come from an external retrieved pool; generative synthesis is disallowed for VIG.
    Problem formulation hard constraint ∀ik∈A, ik∈I_retrieved; defines the task vs pure generation.
  • domain assumption GRPO with the stated composite reward improves the agent’s search–insert policy relative to SFT and zero-shot tool use.
    Core training hypothesis; supported empirically but not derived from first principles.
  • domain assumption MRAMG-Bench gold interleaved answers and ID sets are a valid external standard for selection, placement quality (via GPT-4o C.S.), and order.
    All main claims are benchmarked against this suite’s annotations and judge protocol.
  • domain assumption Context-anchored text-to-caption retrieval is an adequate surrogate for finding the right figures in knowledge-intensive corpora.
    Retrieval strategy section; ablated vs CLIP-style matching.
  • standard math Standard ReAct loop semantics and sequence modeling / RL optimization math hold for the MLLM policy.
    Background from Yao et al. 2022 and Shao et al. 2024 GRPO; used without modification proofs.
invented entities (3)
  • Verified Image Grounding (VIG) task framing independent evidence
    purpose: Name and constrain interleaved generation to retrieved authentic images with placement decisions.
    Partially shared with prior VIG/MRAMG work the paper cites; sharpened here as agentic search–selection–insertion rather than a wholly new physical object.
  • Gated composite VIG reward (format × text-judge + image-precision + search-recall) no independent evidence
    purpose: Supervise trajectories without pure EM sparsity or full MLLM-judge hacking on image IDs.
    Paper-specific reward design; independent handle is ablation tables showing collapse under EM-only or MLLM-only alternatives.
  • Symbolic visual referencing (<img:id> tags) independent evidence
    purpose: Let the LLM place images without emitting pixels, preserving provenance.
    Engineering mechanism common in tool-use systems; not a new scientific entity, but central to the method’s authenticity guarantee.

pith-pipeline@v1.2.0-daily-grok45 · 25796 in / 3883 out tokens · 79372 ms · 2026-07-31T19:13:39.651907+00:00 · methodology

0 comments
read the original abstract

In knowledge-intensive scenarios, providing reliable interleaved text-image responses requires Verified Image Grounding (VIG): the precise integration of retrieved authentic visual evidence. Existing retrieval-augmented frameworks predominantly rely on decoupled, static pipelines, inherently failing to dynamically reason about when external knowledge is required and where visual assets should be contextually inserted. To bridge this gap, we propose VIG-RL, an autonomous agentic framework that formulates the search-selection-insertion workflow as an active decision-making process. Operating within a dynamic ReAct-style loop, VIG-RL is optimized via reinforcement learning, guided by a composite reward system that holistically evaluates the agent's step-by-step tool execution and final multimodal alignment. Extensive evaluations demonstrate that VIG-RL establishes a new state-of-the-art, significantly outperforming existing static baselines.

Figures

Figures reproduced from arXiv: 2607.28055 by Chong Chen, Jun Guang, Qinhan Yu, Wentao Zhang.

Figure 1
Figure 1. Figure 1: Verified Image Grounding (VIG) vs. Pure Gen￾eration. The model-generated image of the Giant Wild Goose Pagoda contains architectural inconsistencies, includ￾ing an incorrect number of stories, wrong railing colors, and missing structural details near the arch openings. In con￾trast, VIG retrieves authentic photographs, yielding factually grounded and visually consistent multimodal responses. synthetic appr… view at source ↗
Figure 2
Figure 2. Figure 2: Illustration of the VIG-RL workflow. Given a user question, the agent iteratively reasons, performs text/image search, [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Comparison of the zero-shot agentic baseline [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figure 5
Figure 5. Figure 5: Agentic retrieval and response quality on [PITH_FULL_IMAGE:figures/full_fig_p006_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: Comparison of EM and model-judge-only re [PITH_FULL_IMAGE:figures/full_fig_p007_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: Case study on the Manual dataset. VIG-RL-8B retrieves both textual evidence and verified figures from the source manual and inserts the figures at the corresponding procedural steps, demonstrating its ability to jointly perform evidence retrieval, image selection, and text-image alignment [PITH_FULL_IMAGE:figures/full_fig_p017_7.png] view at source ↗
Figure 8
Figure 8. Figure 8: Case of Qwen3-VL-8B-Instruct under the agentic setting on the [PITH_FULL_IMAGE:figures/full_fig_p018_8.png] view at source ↗
Figure 9
Figure 9. Figure 9: Case of Gemini-2.5-Flash under the agentic setting on the [PITH_FULL_IMAGE:figures/full_fig_p019_9.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

58 extracted references · 23 linked inside Pith

  1. [1]

    International Conference on Learning Representations , volume=

    Interleaved scene graphs for interleaved text-and-image generation assessment , author=. International Conference on Learning Representations , volume=

  2. [3]

    International Conference on Learning Representations , volume=

    Dreamllm: Synergistic multimodal comprehension and creation , author=. International Conference on Learning Representations , volume=

  3. [10]

    Proceedings of the 48th International ACM SIGIR Conference on Research and Development in Information Retrieval , pages=

    MRAMG-Bench: a comprehensive benchmark for advancing multimodal retrieval-augmented multimodal generation , author=. Proceedings of the 48th International ACM SIGIR Conference on Research and Development in Information Retrieval , pages=

  4. [12]

    2024 , journal =

    HybridFlow: A Flexible and Efficient RLHF Framework , author =. 2024 , journal =

    2024

  5. [14]

    2024 , howpublished =

    GPT-4o , author =. 2024 , howpublished =

    2024

  6. [16]

    Advances in neural information processing systems , volume=

    Language models are few-shot learners , author=. Advances in neural information processing systems , volume=

  7. [19]

    DeepSeek-V4: Towards Highly Efficient Million-Token Context Intelligence , author=

  8. [20]

    National Science Review , volume=

    A survey on multimodal large language models , author=. National Science Review , volume=. 2024 , publisher=

    2024

  9. [24]

    International Conference on Learning Representations , volume=

    Show-o: One single transformer to unify multimodal understanding and generation , author=. International Conference on Learning Representations , volume=

  10. [25]

    International Conference on Learning Representations , volume=

    Transfusion: Predict the next token and diffuse images with one multi-modal model , author=. International Conference on Learning Representations , volume=

  11. [28]

    arXiv preprint arXiv:2509.13642 , year=

    LLM-I: LLMs are naturally interleaved multimodal creators , author=. arXiv preprint arXiv:2509.13642 , year=

    arXiv

  12. [30]

    Advances in Neural Information Processing Systems , volume=

    Generating images with multimodal language models , author=. Advances in Neural Information Processing Systems , volume=

  13. [31]

    Forty-first International Conference on Machine Learning , year=

    Next-gpt: Any-to-any multimodal llm , author=. Forty-first International Conference on Machine Learning , year=

  14. [32]

    International Conference on Machine Learning , pages=

    Grounding language models to images for multimodal inputs and outputs , author=. International Conference on Machine Learning , pages=. 2023 , organization=

    2023

  15. [33]

    Proceedings of the 31st International Conference on Computational Linguistics: System Demonstrations , pages=

    Murar: A simple and effective multimodal retrieval and answer refinement framework for multimodal question answering , author=. Proceedings of the 31st International Conference on Computational Linguistics: System Demonstrations , pages=

  16. [35]

    Findings of the Association for Computational Linguistics: ACL 2024 , pages=

    TextBind: Multi-turn interleaved multimodal instruction-following in the wild , author=. Findings of the Association for Computational Linguistics: ACL 2024 , pages=

    2024

  17. [36]

    arXiv preprint arXiv:2309.15112 , year=

    Internlm-xcomposer: A vision-language large model for advanced text-image comprehension and composition , author=. arXiv preprint arXiv:2309.15112 , year=

    Pith/arXiv arXiv

  18. [37]

    arXiv preprint arXiv:2401.10208 , year=

    Mm-interleaved: Interleaved image-text generative modeling via multi-modal feature synchronizer , author=. arXiv preprint arXiv:2401.10208 , year=

    Pith/arXiv arXiv

  19. [40]

    Advances in Neural Information Processing Systems , volume=

    Vrag-rl: Empower vision-perception-based rag for visually rich information understanding via iterative reasoning with reinforcement learning , author=. Advances in Neural Information Processing Systems , volume=

  20. [41]

    2024 , eprint=

    BGE M3-Embedding: Multi-Lingual, Multi-Functionality, Multi-Granularity Text Embeddings Through Self-Knowledge Distillation , author=. 2024 , eprint=

    2024

  21. [42]

    Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 3: System Demonstrations) , address=

    LlamaFactory: Unified Efficient Fine-Tuning of 100+ Language Models , author=. Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 3: System Demonstrations) , address=. 2024 , url=

    2024

  22. [43]

    OpenAI Blog , year=

    Hello GPT-4o , author=. OpenAI Blog , year=

  23. [44]

    Bai, S.; Cai, Y.; Chen, R.; Chen, K.; Chen, X.; Cheng, Z.; Deng, L.; Ding, W.; Gao, C.; Ge, C.; et al. 2025. Qwen3-vl technical report. arXiv preprint arXiv:2511.21631

    Pith/arXiv arXiv 2025

  24. [45]

    D.; Dhariwal, P.; Neelakantan, A.; Shyam, P.; Sastry, G.; Askell, A.; et al

    Brown, T.; Mann, B.; Ryder, N.; Subbiah, M.; Kaplan, J. D.; Dhariwal, P.; Neelakantan, A.; Shyam, P.; Sastry, G.; Askell, A.; et al. 2020. Language models are few-shot learners. Advances in neural information processing systems, 33: 1877--1901

    2020

  25. [46]

    Chen, J.; Xiao, S.; Zhang, P.; Luo, K.; Lian, D.; and Liu, Z. 2024. BGE M3-Embedding: Multi-Lingual, Multi-Functionality, Multi-Granularity Text Embeddings Through Self-Knowledge Distillation. arXiv:2402.03216

    Pith/arXiv arXiv 2024

  26. [47]

    Chen, X.; Wu, Z.; Liu, X.; Pan, Z.; Liu, W.; Xie, Z.; Yu, X.; and Ruan, C. 2025. Janus-pro: Unified multimodal understanding and generation with data and model scaling. arXiv preprint arXiv:2501.17811

    Pith/arXiv arXiv 2025

  27. [48]

    Chern, E.; Su, J.; Ma, Y.; and Liu, P. 2024. Anole: An open, autoregressive, native large multimodal models for interleaved image-text generation. arXiv preprint arXiv:2407.06135

    Pith/arXiv arXiv 2024

  28. [49]

    Comanici, G.; Bieber, E.; Schaekermann, M.; Pasupat, I.; Sachdeva, N.; Dhillon, I.; Blistein, M.; Ram, O.; Zhang, D.; Rosen, E.; et al. 2025. Gemini 2.5: Pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capabilities. arXiv preprint arXiv:2507.06261

    Pith/arXiv arXiv 2025

  29. [50]

    Dong, R.; Peng, Y.; Qi, Z.; Ge, Z.; Yang, J.; Zhao, L.; Sun, J.; Zhou, H.; Wei, H.; Kong, X.; et al. 2024. Dreamllm: Synergistic multimodal comprehension and creation. In International Conference on Learning Representations, volume 2024, 6666--6702

    2024

  30. [51]

    Guo, D.; Yang, D.; Zhang, H.; Song, J.; Zhang, R.; Xu, R.; Zhu, Q.; Ma, S.; Wang, P.; Bi, X.; et al. 2025. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501.12948

    Pith/arXiv arXiv 2025

  31. [52]

    Hong, J.; Zhao, C.; Zhu, C.; Lu, W.; Xu, G.; and Yu, X. 2025. Deepeyesv2: Toward agentic multimodal model. arXiv preprint arXiv:2511.05271

    Pith/arXiv arXiv 2025

  32. [53]

    Huang, W.; Zeng, Y.; Wang, Q.; Fang, Z.; Cao, S.; Chu, Z.; Yin, Q.; Chen, S.; Yin, Z.; Chen, L.; et al. 2026. Vision-deepresearch: Incentivizing deepresearch capability in multimodal large language models. arXiv preprint arXiv:2601.22060

    arXiv 2026

  33. [54]

    P.; Perelman, A.; Ramesh, A.; Clark, A.; Ostrow, A.; Welihinda, A.; Hayes, A.; Radford, A.; et al

    Hurst, A.; Lerer, A.; Goucher, A. P.; Perelman, A.; Ramesh, A.; Clark, A.; Ostrow, A.; Welihinda, A.; Hayes, A.; Radford, A.; et al. 2024. Gpt-4o system card. arXiv preprint arXiv:2410.21276

    Pith/arXiv arXiv 2024

  34. [55]

    Jaech, A.; Kalai, A.; Lerer, A.; Richardson, A.; El-Kishky, A.; Low, A.; Helyar, A.; Madry, A.; Beutel, A.; Carney, A.; et al. 2024. Openai o1 system card. arXiv preprint arXiv:2412.16720

    Pith/arXiv arXiv 2024

  35. [56]

    Jin, B.; Zeng, H.; Yue, Z.; Yoon, J.; Arik, S.; Wang, D.; Zamani, H.; and Han, J. 2025. Search-r1: Training llms to reason and leverage search engines with reinforcement learning. arXiv preprint arXiv:2503.09516

    Pith/arXiv arXiv 2025

  36. [57]

    Y.; Fried, D.; and Salakhutdinov, R

    Koh, J. Y.; Fried, D.; and Salakhutdinov, R. R. 2023. Generating images with multimodal language models. Advances in Neural Information Processing Systems, 36: 21487--21506

    2023

  37. [58]

    Liu, A.; Feng, B.; Xue, B.; Wang, B.; Wu, B.; Lu, C.; Zhao, C.; Deng, C.; Zhang, C.; Ruan, C.; et al. 2024. Deepseek-v3 technical report. arXiv preprint arXiv:2412.19437

    Pith/arXiv arXiv 2024

  38. [59]

    Ma, Z.-A.; Lan, T.; Tu, R.-C.; Hu, Y.; Zhu, Y.-S.; Zhang, T.; Huang, H.; Wu, Z.; and Mao, X.-L. 2024. Multi-modal retrieval augmented multi-modal generation: Datasets, evaluation metrics and strong baselines. arXiv preprint arXiv:2411.16365

    Pith/arXiv arXiv 2024

  39. [60]

    OpenAI. 2024. Hello GPT-4o. OpenAI Blog

    2024

  40. [61]

    OpenAI. 2025. Introducing GPT-5. https://openai.com/index/introducing-gpt-5/

    2025

  41. [62]

    Shao, Z.; Wang, P.; Zhu, Q.; Xu, R.; Song, J.; Bi, X.; Zhang, H.; Zhang, M.; Li, Y.; Wu, Y.; et al. 2024. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.03300

    Pith/arXiv arXiv 2024

  42. [63]

    Sheng, G.; Zhang, C.; Ye, Z.; Wu, X.; Zhang, W.; Zhang, R.; Peng, Y.; Lin, H.; and Wu, C. 2024. HybridFlow: A Flexible and Efficient RLHF Framework. arXiv preprint arXiv: 2409.19256

    Pith/arXiv arXiv 2024

  43. [64]

    Shi, M.; Zeng, X.; Huang, J.; Cui, Y.; Ferroni, F.; Li, J.; Pachori, S.; Li, Z.; Balaji, Y.; Wang, H.; et al. 2026. DuoGen: Towards General Purpose Interleaved Multimodal Generation. arXiv preprint arXiv:2602.00508

    arXiv 2026

  44. [65]

    X.; Fang, L.; and Wen, J.-R

    Song, H.; Jiang, J.; Min, Y.; Chen, J.; Chen, Z.; Zhao, W. X.; Fang, L.; and Wen, J.-R. 2025. R1-searcher: Incentivizing the search capability in llms via reinforcement learning. arXiv preprint arXiv:2503.05592

    Pith/arXiv arXiv 2025

  45. [66]

    Touvron, H.; Lavril, T.; Izacard, G.; Martinet, X.; Lachaux, M.-A.; Lacroix, T.; Rozi \`e re, B.; Goyal, N.; Hambro, E.; Azhar, F.; et al. 2023. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971

    Pith/arXiv arXiv 2023

  46. [67]

    Wang, Q.; Ding, R.; Zeng, Y.; Chen, Z.; Chen, L.; Wang, S.; Xie, P.; Huang, F.; and Zhao, F. 2025. Vrag-rl: Empower vision-perception-based rag for visually rich information understanding via iterative reasoning with reinforcement learning. Advances in Neural Information Processing Systems, 38: 57133--57160

    2025

  47. [68]

    Wang, X.; Zhang, X.; Luo, Z.; Sun, Q.; Cui, Y.; Wang, J.; Zhang, F.; Wang, Y.; Li, Z.; Yu, Q.; et al. 2024. Emu3: Next-token prediction is all you need. arXiv preprint arXiv:2409.18869

    Pith/arXiv arXiv 2024

  48. [69]

    Wu, J.; Deng, Z.; Li, W.; Liu, Y.; You, B.; Li, B.; Ma, Z.; and Liu, Z. 2025. Mmsearch-r1: Incentivizing lmms to search. arXiv preprint arXiv:2506.20670

    Pith/arXiv arXiv 2025

  49. [70]

    Wu, S.; Fei, H.; Qu, L.; Ji, W.; and Chua, T.-S. 2024. Next-gpt: Any-to-any multimodal llm. In Forty-first International Conference on Machine Learning

    2024

  50. [71]

    Xiao, Z.; Yu, Q.; Li, B.; Chen, G.; Chen, C.; and Zhang, W. 2025. M2io-r1: An efficient rl-enhanced reasoning framework for multimodal retrieval augmented multimodal generation. arXiv preprint arXiv:2508.06328

    Pith/arXiv arXiv 2025

  51. [72]

    J.; Wang, W.; Lin, K

    Xie, J.; Mao, W.; Bai, Z.; Zhang, D. J.; Wang, W.; Lin, K. Q.; Gu, Y.; Chen, Z.; Yang, Z.; and Shou, M. Z. 2025. Show-o: One single transformer to unify multimodal understanding and generation. In International Conference on Learning Representations, volume 2025, 28240--28264

    2025

  52. [73]

    Yao, S.; Zhao, J.; Yu, D.; Du, N.; Shafran, I.; Narasimhan, K.; and Cao, Y. 2022. React: Synergizing reasoning and acting in language models. arXiv preprint arXiv:2210.03629

    Pith/arXiv arXiv 2022

  53. [74]

    Yin, S.; Fu, C.; Zhao, S.; Li, K.; Sun, X.; Xu, T.; and Chen, E. 2024. A survey on multimodal large language models. National Science Review, 11(12): nwae403

    2024

  54. [75]

    Yu, Q.; Xiao, Z.; Li, B.; Wang, Z.; Chen, C.; and Zhang, W. 2025. MRAMG-Bench: a comprehensive benchmark for advancing multimodal retrieval-augmented multimodal generation. In Proceedings of the 48th International ACM SIGIR Conference on Research and Development in Information Retrieval, 3616--3626

    2025

  55. [76]

    Zheng, Y.; Zhang, R.; Zhang, J.; Ye, Y.; Luo, Z.; Feng, Z.; and Ma, Y. 2024. LlamaFactory: Unified Efficient Fine-Tuning of 100+ Language Models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 3: System Demonstrations). Bangkok, Thailand: Association for Computational Linguistics

    2024

  56. [77]

    Zhou, C.; Yu, L.; Babu, A.; Tirumala, K.; Yasunaga, M.; Shamis, L.; Kahn, J.; Ma, X.; Zettlemoyer, L.; and Levy, O. 2025. Transfusion: Predict the next token and diffuse images with one multi-modal model. In International Conference on Learning Representations, volume 2025, 6446--6469

    2025

  57. [78]

    J.; Lian, D.; and Xiong, Y

    Zhou, J.; Liu, Z.; Liu, Z.; Xiao, S.; Wang, Y.; Zhao, B.; Zhang, C. J.; Lian, D.; and Xiong, Y. 2024. MegaPairs: Massive Data Synthesis For Universal Multimodal Retrieval. arXiv preprint arXiv:2412.14475

    Pith/arXiv arXiv 2024

  58. [79]

    S.; Feujio, L.; Maharaj, A.; and Li, Y

    Zhu, Z.; Lee, D.; Zhang, H.; Harsha, S. S.; Feujio, L.; Maharaj, A.; and Li, Y. 2025. Murar: A simple and effective multimodal retrieval and answer refinement framework for multimodal question answering. In Proceedings of the 31st International Conference on Computational Linguistics: System Demonstrations, 126--135

    2025