Pith. sign in

REVIEW 4 major objections 6 minor 57 references

M$^3$Prune: Hierarchical Collaborative Pruning for Efficient Multi-Modal Multi-Agent Retrieval-Augmented Generation

T0 review · 4 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read M3Prune claims that pruning redundant agent-to-agent links in multi-modal multi-agent retrieval-augmented generation improves both accuracy and token efficiency, reporting 9.4% higher accuracy and 23.8% better token efficiency than strong…

desk verdict The pruning idea is plausible and the system is well built, but the headline accuracy/token gains are not yet pinned to the pruning mechanism because no same-system-without-pruning control is reported. read the letter →

arxiv 2608.05967 v1 pith:GBLLO7PR submitted 2026-08-06 cs.MM

Taolin Zhang , Weizi shao , Zijie Zhou , Chen Chen , Daiyang Yu , Tingyuan Hu , Chengyu Wang , Xiaofeng He This is my paper · ORCID
classification cs.MM
keywords multi-modalretrieval-augmentedgenerationmulti-agentsystemscommunicationgraphpruningtokenefficiencysparsificationvisualquestionansweringspatio-temporalmessagepassing
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tries to show that multi-agent multi-modal retrieval-augmented generation (mRAG) suffers from redundant agent-to-agent communication, and that systematically pruning the communication graph improves both accuracy and token cost. It proposes M3Prune, which learns sparsified spatio-temporal graphs within each modality, then across modalities, with a modality-alignment regularizer, and progressively removes edges. On Vidoseek, MultimodalQA, and ScienceQA it reports state-of-the-art accuracy among the compared methods, with a 9.4% accuracy gain and a 23.8% token-efficiency gain over strong multi-agent baselines. If true, communication topology itself is a learnable resource: deciding which agents talk to whom matters as much as what they say.

What carries the argument

The central machinery is a pair of spatio-temporal communication graphs: one per modality (textual and visual) and one inter-modal graph that joins them. Edge weights are learnable softened adjacency matrices, trained by a policy-gradient estimate of task utility with nuclear-norm sparsity regularization and a modality alignment score, then progressively pruned via top-K masks at a decaying rate. This lets the framework route messages only along task-critical links, both within and across modalities.

What would settle it

Retrain M3Prune on independent splits with 20, 40, 80, and 160 examples and different random seeds, then compare the learned pruned graphs and the accuracy gains over fixed-topology baselines. If the gains vanish at larger training sizes or the surviving edges vary wildly across seeds, the learned communication graph is not the cause. A crisper check is to give the best fixed-topology baseline the same token budget as M3Prune; if it matches accuracy per token, the pruning claim collapses.

Watch

Extended reading notes

Core claim

The central claim is that communication redundancy, not agent capability, is the limiting factor in multi-agent mRAG, and that pruning redundant edges is a learnable optimization problem. M3Prune models each modality as a graph with spatial edges (same-round messages) and temporal edges (memory across rounds), initializes softened adjacency matrices with Gumbel-Softmax, optimizes expected task utility via policy gradient with nuclear-norm sparsity, aligns cross-modal edge patterns with a cosine alignment score, and progressively prunes low-weight edges. The resulting sparse hierarchical topology is then frozen for inference, yielding higher accuracy with fewer tokens than fixed-topology multi-agent systems.

Load-bearing premise

The method's gains rest on 40 training examples per stage being enough for the policy-gradient estimator to learn a communication graph that generalizes, and on the DeepSeek-V3 judge scoring M3Prune's answers without favoring them.

Editorial extensions

If this is right

  • The learned pruned communication graph can replace a fixed full-connectivity topology at inference time, so the efficiency gain is built into the deployed system rather than requiring per-query search.
  • Ablations indicate that the inter-modal sparsification stage and the modality alignment score each contribute independently to accuracy, so removing either degrades performance.
  • The reported token-efficiency improvement means the method achieves higher accuracy while consuming fewer tokens than strong fixed-topology multi-agent baselines on the tested benchmarks.
  • Adversarial robustness results suggest that pruning and reweighting edges can reduce the influence of compromised or misleading agents, a property that fixed-topology systems lack.
  • The framework transfers across three backbone models of different scales, from 7B open-source models to a closed-source MLLM, indicating the pruning mechanism is not tied to one backbone.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Because the learned topology is trained on only 40 examples per stage, a natural extension is to test whether pruned graphs transfer across datasets and backbones; the paper's choice to train on MultimodalQA for Vidoseek suggests transfer is expected but not isolated.
  • The same REINFORCE plus Gumbel-Softmax machinery should extend to more than two modalities or to heterogeneous agent types, since the hierarchical two-stage design is a special case of a larger communication graph.
  • Pruning is frozen at training time, so the cost saving is static; an adaptive per-query pruning policy could push token efficiency further, but that is beyond what the paper demonstrates.
  • The reliance on a single LLM judge for semantic accuracy means the reported gains should be checked against multiple judges or human evaluation before treating the 9.4% figure as model-independent.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper proposes M3Prune, a hierarchical communication-graph pruning framework for multi-modal multi-agent retrieval-augmented generation (mRAG). It learns softened adjacency matrices for intra-modal and inter-modal agent communication graphs using Gumbel-Softmax initialization and a policy-gradient objective, introduces a modality alignment score to encourage cross-modal consistency, and progressively prunes edges during training to obtain a compact inference topology. Experiments on ScienceQA, Vidoseek, and MultimodalQA with three backbone models report state-of-the-art accuracy and improved token efficiency over single-agent and multi-agent baselines, with a claimed 9.4% accuracy improvement and 23.8% token-efficiency improvement.

Significance. If the central claim is sustained, M3Prune would be a practical contribution to the deployment of multi-agent mRAG systems, where token overhead is a known bottleneck. The paper has notable strengths: it releases code and data, evaluates on three benchmarks covering general and domain-specific settings, reports statistical significance claims, includes a token-efficiency analysis, and tests robustness under adversarial agent perturbations. However, the current evidence does not isolate the pruning mechanism from the rest of the multi-agent pipeline, and the training protocol is exceptionally small-scale (40 instances per stage), so the headline gains cannot yet be attributed to hierarchical pruning. The contribution is potentially useful but needs substantially stronger controlled evaluation.

major comments (4)
  1. [Section 5.1, Implementation Details] The communication graph is trained with only 40 instances per stage, using the policy-gradient estimator of Eq. 7 with K=10 sampled graphs. The paper provides no learning curves, no validation split, and no demonstration that the learned topology stabilizes or generalizes rather than overfitting to the 40 training examples. Appendix Table 7 varies the number of training samples (#20/#40/#80) and random seeds, but it reports only final test accuracy and does not include any topology-control condition. Because the central claim is that the learned pruned graph transfers to test data, this is a load-bearing gap. Please add learning curves, multiple-seed variance, and a train/validation split for early stopping, or otherwise justify the 40-instance protocol.
  2. [Table 4 and Section 5.3] The ablations remove entire modules (w/o Lalign, w/o norm, w/o Ginter, w/o Gintra-txt, w/o Gintra-vis) but never compare M3Prune against the identical agent prompts, backbone, and summary module using a fixed fully connected communication graph, nor against a random sparse graph matched to the same edge count and token budget. Without these controls, the 9.4% accuracy and 23.8% token-efficiency improvements over multi-agent baselines cannot be attributed to the pruning mechanism; they could stem from the multi-agent prompt design, the summary module, or the specific agent roles. This control is essential for the paper's title and main claim and should be added.
  3. [Eq. (6), Eq. (11), Definition 1] The utility function phi(·) is never specified. The objective in Eq. 6 and Eq. 11 maximizes expected phi(G), and Eq. 7 estimates its gradient from sampled graphs, but the manuscript does not define phi operationally, how it is computed for a single sampled graph, or whether the same phi is used for intra- and inter-modal stages. If phi is the DeepSeek-V3 judge score mentioned in Section 5.1, then the reward is stochastic and non-deterministic, and the policy-gradient estimator with no baseline subtraction, K=10 samples, and 40 training instances becomes high-variance. Please specify phi precisely, report its variance, and provide a judge-audit or agreement analysis.
  4. [Section 4.3 and Section 5.1 (Baselines)] The method explicitly follows the pruning-rate schedule and policy-gradient approach of Cut the Crap [48], yet [48] is not included as a baseline in Tables 1 or 2. Without comparing against the direct predecessor adapted to the same multi-modal setting, the marginal contribution of the hierarchical design, modality alignment, and multi-modal extension cannot be assessed. Additionally, footnote 3 states that Vidoseek is evaluated using a topology trained on MultimodalQA's training set; the paper does not analyze how this cross-dataset transfer affects the Vidoseek results or whether the learned graph from MultimodalQA is appropriate for Vidoseek. Both issues should be addressed.
minor comments (6)
  1. [Eq. (13)] The notation TopK(X, K) overloads K, which is already used for the number of sampled graphs in Eq. (7) and Algorithm 1. Please use a different symbol, e.g., k_top, for the top-k selection parameter.
  2. [Table 5] The notation table lists V_joint and S_joint, but Section 3 defines the joint node and memory sets as V and S. Please unify the notation.
  3. [Section 5.4 and Appendix C.3] The text says 'Response Attack' while the appendix and Figure 16 say 'Response Prompt Attack'. Please make the terminology consistent.
  4. [References] Reference [43] is cited for Qwen-VL-Max but is titled 'Qwen3-Omni Technical Report'. Please verify that this is the correct reference for the closed-source model used, or replace it with the appropriate Qwen-VL-Max citation.
  5. [References] References [25] and [26] appear to be the same paper (Lu et al., 'Learn to Explain') with different venue details. Please merge or disambiguate them.
  6. [Tables 1 and 2] The tables state that t-tests demonstrate statistical significance at p<0.05, but no details are given about the number of runs, how seeds were handled, or how the t-test was computed across the reported ± values. Please describe the statistical testing procedure.

Circularity Check

0 steps flagged · score 1.0 of 10

No circular derivation: the learned pruning graphs are optimized on held-out training instances and evaluated on held-out test sets against external multi-agent baselines; the one self-citation ([50]) is contextual and not load-bearing.

full rationale

Walking the derivation chain, the central claim does not reduce to its inputs by construction. The softened adjacency matrices are initialized with Gumbel-Softmax noise (Eq. 3), optimized with a policy-gradient estimator (Eq. 7) whose reward is task utility φ on sampled graphs, regularized by nuclear-norm sparsity (Eq. 6) and a modality-alignment score (Eq. 12), and finally thresholded by a Top-K pruning mask (Eq. 13). No equation defines the predicted accuracy as the fitted objective; every evaluation reported in Tables 1–2 is on the official test splits of MultimodalQA, Vidoseek, and ScienceQA, while graph training uses 40 training instances per stage. Comparisons are against external methods (OmniSearch, ViDoRAG, HM-RAG, E-Agent, single-agent RAG systems), not against the paper's own fitted values renamed as predictions. The only self-citation, BELLE [50], appears in a general related-work sentence about textual RAG and is not used to justify the pruning mechanism or to forbid alternative designs; the progressive-pruning schedule is borrowed from the non-overlapping Cut the Crap work [48]. A genuinely missing experiment is a same-pipeline no-pruning control that would isolate the contribution of the learned topology, and the DeepSeek-V3 judge is used both as a training reward and as the Acc* evaluation metric; both are validity and attribution concerns, not circularity, because the train/test separation prevents the test numbers from being forced by construction.

Assumptions & free parameters 7 free parameters · 7 assumptions · 0 invented entities

The framework relies on standard ML machinery (Gumbel-Softmax, REINFORCE, nuclear-norm regularization) plus several domain-specific assumptions that are only supported by ablations. The most consequential assumptions are that 40 training examples are sufficient and that the DeepSeek-V3 judge is unbiased; neither is demonstrated outside the paper's own experiments.

free parameters (7)
  • Number of training instances per stage = 40
    Chosen in Section 5.1; no sensitivity analysis in the main text.
  • Learning rate eta = 0.1
    Set in Section 5.1, no schedule described.
  • Noise tolerance delta = 0.1
    Selected in Section 5.1; Appendix C.2 varies it, indicating it is not a theoretically fixed constant.
  • Initial pruning rate p(0) = not explicitly reported
    Pruning-rate schedule in Eq. 13 requires p(0); Appendix C.2 sweeps intra-modal and inter-modal rates, so p(0) is a fitted hyperparameter.
  • Number of agents per modality = 4 or 5 depending on dataset
    Selected empirically per dataset (Section 5.1).
  • Number of communication rounds T = 2
    Set in Section 5.1; Table 6 shows performance varies with T, confirming it is a tuned hyperparameter.
  • Number of sampled graphs K = 10
    Set in Section 5.1; no sensitivity analysis reported.
assumptions (7)
  • standard math Gumbel-Softmax provides differentiable approximations to discrete edge sampling (Eq. 3).
    Standard reparameterization trick; assumed without proof.
  • standard math Policy gradient with K=10 samples gives unbiased gradient estimates usable with MLLM API rewards (Eq. 7).
    Standard REINFORCE; in practice with high variance it may fail to learn from 40 samples.
  • domain assumption Nuclear-norm regularization on softened adjacency matrices induces useful sparsity (Eq. 6 and Eq. 11).
    No proof that this sparsity improves task performance; ablation 'w/o norm' is the only evidence.
  • ad hoc to paper Modality alignment score Lalign between inter-modal adjacency directions improves cross-modal consistency (Eq. 12).
    Introduced specifically for this framework; its effect is only shown via a single ablation.
  • domain assumption DeepSeek-V3 semantic judge yields valid accuracy scores (Acc*) for model outputs (Section 5.1).
    No inter-annotator agreement or validation against human labels reported.
  • ad hoc to paper 40 training instances per stage are sufficient for learning the communication graph (Section 5.1).
    This is a load-bearing capacity assumption; no convergence or learning-curve evidence is provided.
  • domain assumption Using MultimodalQA training data for Vidoseek is acceptable because distributions are similar (Section 5.1).
    No quantitative distributional similarity analysis is provided.

how reviews work

0 comments
Cite this review

Pith. "Pith review of M$^3$Prune: Hierarchical Collaborative Pruning for Efficient Multi-Modal Multi-Agent Retrieval-Augmented Generation." pith.science (2026). https://pith.science/paper/GBLLO7PR

@misc{pith2026260805967,
  author       = {Pith},
  title        = {Pith review of: M$^3$Prune: Hierarchical Collaborative Pruning for Efficient Multi-Modal Multi-Agent Retrieval-Augmented Generation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/GBLLO7PR}},
  note         = {Machine review of arXiv:2608.05967}
}
read the original abstract

Recent advances in multi-modal retrieval-augmented generation (mRAG), which augments multi-modal large language models (MLLMs) with external knowledge, have shown that collective intelligence from multiple agents can outperform a single model through effective communication. Despite their strong performance, existing multi-agent systems incur substantial token overhead and computational cost, posing challenges for large-scale deployment. To address these issues, we propose a Multi-Modal Multi-agent hierarchical communication graph PRUNING framework, termed M3Prune. M3Prune eliminates redundant communication edges both across and within modalities, improving the trade-off between task performance and token overhead. Specifically, M3Prune first performs intra-modal graph sparsification in the textual and visual modalities to identify task-critical communication links. It then constructs an inter-modal communication graph and sparsifies cross-modal connections while encouraging consistent cross-modal reasoning through a modality alignment score. Finally, it progressively prunes redundant edges to obtain an efficient hierarchical topology. Extensive experiments on both general-domain and domain-specific mRAG benchmarks show that M3Prune consistently outperforms single-agent and strong multi-agent mRAG systems while signifi- cantly improving token efficiency.

Figures

Figures reproduced from arXiv: 2608.05967 by the authors.

Figure 1
Figure 1. Comparison of our approach with existing methods. (1) [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Overview. The key components include: (1) [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗
Figure 3
Figure 3. Evolution of communication edge weights from visual-to-text (Left) and text-to-visual (Right) agents on ScienceQA. [PITH_FULL_IMAGE:figures/full_fig_p008_3.png] view at source ↗
Figures from the paper (15 more)
Figure 5
Figure 5. Figure 5: Model performance under adversarial attacks [PITH_FULL_IMAGE:figures/full_fig_p008_5.png]
Figure 6
Figure 6. Figure 6: Prompt template for the Image Critic. Image Data Analyst You will be given a question and relevant context, you need to start from the image context, focus on the visual information in the image, and consider which visual elements would be helpful for answering the que…
Figure 10
Figure 10. Figure 10: Prompt template for the Text Critic. Text Data Analyst You will be given a question and relevant context, you need to consider which information in the provided context would be helpful for answering the question. Please refer to them step by step to give your answer …
Figure 8
Figure 8. Figure 8: Prompt template for the Image Knowledgeable Ex [PITH_FULL_IMAGE:figures/full_fig_p011_8.png]
Figure 12
Figure 12. Figure 12: Prompt template for the Text Knowledge Expert. [PITH_FULL_IMAGE:figures/full_fig_p011_12.png]
Figure 13
Figure 13. Figure 13: Prompt template for the Text Logic Expert. [PITH_FULL_IMAGE:figures/full_fig_p013_13.png]
Figure 15
Figure 15. Figure 15: Prompt template for input prompt attack. [PITH_FULL_IMAGE:figures/full_fig_p013_15.png]
Figure 16
Figure 16. Figure 16: Prompt template for response prompt attack. [PITH_FULL_IMAGE:figures/full_fig_p013_16.png]
Figure 17
Figure 17. Figure 17: The influence of different edge pruning rates using [PITH_FULL_IMAGE:figures/full_fig_p014_17.png]
Figure 18
Figure 18. Figure 18: The influence of different edge pruning rates using [PITH_FULL_IMAGE:figures/full_fig_p014_18.png]
Figure 19
Figure 19. Figure 19: Performance under adversarial attacks, including [PITH_FULL_IMAGE:figures/full_fig_p015_19.png]
Figure 20
Figure 20. Figure 20: Evolution of communication edge weights from visual-to-text (Top) and text-to-visual (Bottom) directions on [PITH_FULL_IMAGE:figures/full_fig_p016_20.png]
Figure 21
Figure 21. Figure 21: Evolution of communication edge weights from visual-to-text (Top) and text-to-visual (Bottom) directions on [PITH_FULL_IMAGE:figures/full_fig_p016_21.png]
Figure 22
Figure 22. Figure 22: Token efficiency comparison for different multi-agent models on Llama3.2-VL-11B. The number of tokens consumed [PITH_FULL_IMAGE:figures/full_fig_p017_22.png]
Figure 23
Figure 23. Figure 23: Token efficiency comparison for different multi-agent models on Qwen2.5-VL-7B. [PITH_FULL_IMAGE:figures/full_fig_p017_23.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

57 extracted references · 39 canonical work pages

  1. [48]

    Guibin Zhang, Yanwei Yue, Zhixun Li, Sukwon Yun, Guancheng Wan, Kun Wang, Dawei Cheng, Jeffrey Xu Yu, and Tianlong Chen. 2025. Cut the Crap: An Economical Communication Pipeline for LLM-based Multi-Agent Systems. In International Conference on Learning Representations

    work page 2025

  2. [1]

    Pravallika Abbineni, Saoud Aldowaish, Colin Liechty, Soroosh Noorzad, Ali Ghazizadeh, and Morteza Fayazi. 2025. MuaLLM: A Multimodal Large Language Model Agent for Circuit Design Assistance with Hybrid Contextual Retrieval- Augmented Generation.CoRRabs/2508.08137 (2025)

    arXiv 2025

  3. [2]

    Mohammad Mahdi Abootorabi, Amirhosein Zobeiri, Mahdi Dehghani, Moham- madali Mohammadkhani, Bardia Mohammadi, Omid Ghahroodi, Mahdieh Soley- mani Baghshah, and Ehsaneddin Asgari. 2025. Ask in Any Modality: A Compre- hensive Survey on Multimodal Retrieval-Augmented Generation. InFindings of the Association for Computational Linguistics. 16776–16809

    work page 2025

  4. [3]

    Arian Askari, Emmanouil Stergiadis, Ilya Gusev, and Moran Beladev. 2025. HotelMatch-LLM: Joint Multi-Task Training of Small and Large Language Mod- els for Efficient Multimodal Hotel Retrieval. InProceedings of the Association for Computational Linguistics. 607–619

    work page 2025

  5. [4]

    Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, et al . 2025. Qwen2. 5-vl technical report.arXiv:2502.13923(2025)

    arXiv 2025

  6. [5]

    Davide Caffagni, Federico Cocchi, Nicholas Moratelli, Sara Sarto, Marcella Cornia, Lorenzo Baraldi, and Rita Cucchiara. 2024. Wiki-LLaVA: Hierarchical Retrieval- Augmented Generation for Multimodal LLMs. InIEEE/CVF Conference on Com- puter Vision and Pattern Recognition. 1818–1826

    work page 2024

  7. [6]

    Qi Cai, Yingwei Pan, Ting Yao, Chong-Wah Ngo, and Tao Mei. 2023. ObjectFu- sion: Multi-modal 3D Object Detection with Object-Centric Fusion. InIEEE/CVF International Conference on Computer Vision. 18021–18030

    work page 2023

  8. [7]

    Guillem Capellera, Antonio Rubio, Luis Ferraz, and Antonio Agudo. 2025. Unified Uncertainty-Aware Diffusion for Multi-Agent Trajectory Modeling. InIEEE/CVF Conference on Computer Vision and Pattern Recognition. 22476–22486

    work page 2025

Show all 57 references
  1. [8]

    Chia-Yuan Chang, Zhimeng Jiang, Vineeth Rakesh, Menghai Pan, Chin- Chia Michael Yeh, Guanchu Wang, Mingzhi Hu, Zhichao Xu, Yan Zheng, Ma- hashweta Das, and Na Zou. 2025. MAIN-RAG: Multi-Agent Filtering Retrieval- Augmented Generation. InProceedings of the Association for Compu...

    2025

  2. [9]

    Jiawei Chen, Yue Jiang, Dingkang Yang, Mingcheng Li, Jinjie Wei, Ziyun Qian, and Lihua Zhang. 2024. Can LLMs’ Tuning Methods Work in Medical Multimodal Domain?. InMedical Image Computing and Computer Assisted Intervention. 112– 122

    2024

  3. [10]

    Sanghyuk Chun, Seong Joon Oh, Rafael Sampaio de Rezende, Yannis Kalantidis, and Diane Larlus. 2021. Probabilistic Embeddings for Cross-Modal Retrieval. In IEEE Conference on Computer Vision and Pattern Recognition. 8415–8424

    2021

  4. [11]

    Guanting Dong, Chenghao Zhang, Mengjie Deng, Yutao Zhu, Zhicheng Dou, and Ji-Rong Wen. 2025. Progressive Multimodal Reasoning via Active Retrieval. In Proceedings of the Association for Computational Linguistics. 3579–3602

    2025

  5. [12]

    Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. 2024. The llama 3 herd of models.arXiv:2407.21783(2024)

    2024 arXiv

  6. [13]

    Yushi Hu, Benlin Liu, Jungo Kasai, Yizhong Wang, Mari Ostendorf, Ranjay Kr- ishna, and Noah A. Smith. 2023. TIFA: Accurate and Interpretable Text-to-Image Faithfulness Evaluation with Question Answering. InIEEE/CVF International Conference on Computer Vision. 20349–20360

    2023

  7. [14]

    Eric Jang, Shixiang Gu, and Ben Poole. 2016. Categorical reparameterization with gumbel-softmax.arXiv:1611.01144(2016)

    2016 arXiv

  8. [15]

    Su, Camillo Jose Taylor, and Tanwi Mallick

    Bowen Jiang, Yangxinyu Xie, Xiaomeng Wang, Yuan Yuan, Zhuoqun Hao, Xinyi Bai, Weijie J. Su, Camillo Jose Taylor, and Tanwi Mallick. 2025. Towards Ra- tionality in Language and Multimodal Agents: A Survey. InProceedings of the Nations of the Americas Chapter of the Association ...

    2025

  9. [16]

    Aisha Urooj Khan, Hilde Kuehne, Kevin Duarte, Chuang Gan, Niels da Vito- ria Lobo, and Mubarak Shah. 2021. Found a Reason for me? Weakly-supervised Grounded Visual Question Answering using Capsules. InIEEE Conference on Computer Vision and Pattern Recognition. 8465–8474

    2021

  10. [17]

    Jungsoo Lee, Janghoon Cho, Hyojin Park, Munawar Hayat, Kyuwoong Hwang, Fatih Porikli, and Sungha Choi. 2025. Generalized Contrastive Learning for Universal Multimodal Retrieval.CoRRabs/2509.25638 (2025)

    2025

  11. [18]

    Ioannidis, Huzefa Rangwala, and Christos Faloutsos

    Meng-Chieh Lee, Qi Zhu, Costas Mavromatis, Zhen Han, Soji Adeshina, Vassilis N. Ioannidis, Huzefa Rangwala, and Christos Faloutsos. 2025. HybGRAG: Hybrid Retrieval-Augmented Generation on Textual and Relational Knowledge Bases. In Proceedings of the Association for Computation...

    2025

  12. [19]

    Yangning Li, Yinghui Li, Xinyu Wang, Yong Jiang, Zhen Zhang, Xinran Zheng, Hui Wang, Hai-Tao Zheng, Fei Huang, Jingren Zhou, and Philip S. Yu. 2025. Benchmarking Multimodal Retrieval Augmented Generation with Dynamic VQA Dataset and Self-adaptive Planning Agent. InInternationa...

    2025

  13. [20]

    Junxiong Lin, Yan Wang, Zeng Tao, Boyang Wang, Qing Zhao, Haorang Wang, Xuan Tong, Xinji Mai, Yuxuan Lin, Wei Song, Jiawen Yu, Shaoqi Yan, and Wen- qiang Zhang. 2024. Adaptive Multi-modal Fusion of Spatially Variant Kernel Refinement with Diffusion Model for Blind Image Super-...

    2024

  14. [21]

    Zudi Lin, Erhan Bas, Kunwar Yashraj Singh, Gurumurthy Swaminathan, and Rahul Bhotika. 2023. Relaxing Contrastiveness in Multimodal Representation Learning. InIEEE/CVF Winter Conference on Applications of Computer Vision. 2226–2235. doi:10.1109/WACV56688.2023.00226

    2023

  15. [22]

    Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Cheng- gang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, et al. 2024. Deepseek-v3 technical report.arXiv:2412.19437(2024)

    2024 arXiv

  16. [23]

    Hongjun Liu, Yinghao Zhu, Yuhui Wang, Yitao Long, Zeyu Lai, Lequan Yu, and Chen Zhao. 2025. MedMMV: A Controllable Multimodal Multi-Agent Framework for Reliable and Verifiable Clinical Reasoning.CoRRabs/2509.24314 (2025)

    2025

  17. [24]

    Pei Liu, Xin Liu, Ruoyu Yao, Junming Liu, Siyuan Meng, Ding Wang, and Jun Ma. 2025. HM-RAG: Hierarchical Multi-Agent Multimodal Retrieval Augmented Generation.CoRRabs/2504.12330 (2025)

    2025 arXiv

  18. [25]

    Pan Lu, Swaroop Mishra, Tanglin Xia, Liang Qiu, Kai-Wei Chang, Song-Chun Zhu, Oyvind Tafjord, Peter Clark, and Ashwin Kalyan. 2022. Learn to Explain: MM ’26, November 10–14, 2026, Rio de Janeiro, Brazil Taolin Zhang, Weizi Shao, Zijie Zhou, Chen Chen, Daiyang Yu, Tingyuan Hu, ...

    2022

  19. [26]

    Pan Lu, Swaroop Mishra, Tanglin Xia, Liang Qiu, Kai-Wei Chang, Song-Chun Zhu, Oyvind Tafjord, Peter Clark, and Ashwin Kalyan. 2022. Learn to explain: Multi- modal reasoning via thought chains for science question answering.Advances in Neural Information Processing Systems(2022...

    2022

  20. [27]

    Ziyang Luo, Pu Zhao, Can Xu, Xiubo Geng, Tao Shen, Chongyang Tao, Jing Ma, Qingwei Lin, and Daxin Jiang. 2023. LexLIP: Lexicon-Bottlenecked Language- Image Pre-Training for Large-Scale Image-Text Sparse Retrieval. InIEEE/CVF International Conference on Computer Vision. 11172–11183

    2023

  21. [28]

    Ziyu Ma, Chenhui Gou, Hengcan Shi, Bin Sun, Shutao Li, Hamid Rezatofighi, and Jianfei Cai. 2025. DrVideo: Document Retrieval Based Long Video Understanding. InIEEE/CVF Conference on Computer Vision and Pattern Recognition. 18936–18946

    2025

  22. [29]

    Tianyi Men, Zhuoran Jin, Pengfei Cao, Yubo Chen, Kang Liu, and Jun Zhao

  23. [30]

    Jingwei Peng, Jiehao Chen, Mateo Alejandro Rojas, and Meilin Zhang. 2025. MV- CoRe: Multimodal Visual-Conceptual Reasoning for Complex Visual Question Answering.CoRRabs/2508.07023 (2025)

    2025 arXiv

  24. [31]

    Akhil Perincherry, Jacob Krantz, and Stefan Lee. 2025. Do Visual Imaginations Improve Vision-and-Language Navigation Agents?. InIEEE/CVF Conference on Computer Vision and Pattern Recognition. 3846–3855

    2025

  25. [32]

    Jingyuan Qi, Zhiyang Xu, Rulin Shao, Yang Chen, Di Jin, Yu Cheng, Qifan Wang, and Lifu Huang. 2024. RoRA-VLM: Robust Retrieval-Augmented Vision Language Models.CoRRabs/2410.08876 (2024)

    2024 arXiv

  26. [33]

    Zi Qian, Xin Wang, Xuguang Duan, Pengda Qin, Yuhong Li, and Wenwu Zhu

  27. [34]

    Leonardo Ranaldi, Federico Ranaldi, and Giulia Pucci. 2025. R2-MultiOmnia: Leading Multilingual Multimodal Reasoning via Self-Training. InProceedings of the Association for Computational Linguistics (Volume 1: Long Papers)

    2025

  28. [35]

    Alon Talmor, Ori Yoran, Amnon Catav, Dan Lahav, Yizhong Wang, Akari Asai, Gabriel Ilharco, Hannaneh Hajishirzi, and Jonathan Berant. 2021. Multimodalqa: Complex question answering over text, tables and images.arXiv:2104.06039 (2021)

    2021 arXiv

  29. [36]

    Ryota Tanaka, Taichi Iki, Taku Hasegawa, Kyosuke Nishida, Kuniko Saito, and Jun Suzuki. 2025. VDocRAG: Retrieval-Augmented Generation over Visually-Rich Documents. InIEEE/CVF Conference on Computer Vision and Pattern Recognition. 24827–24837

    2025

  30. [37]

    Yang Tian, Fan Liu, Jingyuan Zhang, Victoria W., Yupeng Hu, and Liqiang Nie

  31. [38]

    Guohua Wang, Shengping Song, Wuchun He, and Yongsen Zheng. 2025. CMHKF: Cross-Modality Heterogeneous Knowledge Fusion for Weakly Supervised Video Anomaly Detection. InProceedings of the Association for Computational Linguistics. 31594–31607

    2025

  32. [39]

    Qiuchen Wang, Ruixue Ding, Zehui Chen, Weiqi Wu, Shihang Wang, Pengjun Xie, and Feng Zhao. 2025. ViDoRAG: Visual Document Retrieval-Augmented Generation via Dynamic Iterative Reasoning Agents.CoRRabs/2502.18017 (2025)

    2025 arXiv

  33. [40]

    InProceedings of the Association for Computational Linguistics

    CoRe-MMRAG: Cross-Source Knowledge Reconciliation for Multimodal RAG. InProceedings of the Association for Computational Linguistics. 32967–32982

  34. [41]

    Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. 2022. Chain-of-thought prompting elicits reasoning in large language models.Advances in neural information processing systems35 (2022), 24824–24837

    2022

  35. [42]

    Ronald J Williams. 1992. Simple statistical gradient-following algorithms for connectionist reinforcement learning.Machine learning8, 3 (1992), 229–256

    1992

  36. [43]

    Yuechen Wang, Yuming Qiao, Dan Meng, Jun Yang, Haonan Lu, Zhenyu Yang, and Xudong Zhang. 2025. Efficient Agent: Optimizing Planning Capability for Multimodal Retrieval Augmented Generation.CoRRabs/2508.08816 (2025)

    2025 arXiv

  37. [44]

    Yibin Yan and Weidi Xie. 2024. EchoSight: Advancing Visual-Language Models with Wiki Knowledge. InFindings of Empirical Methods in Natural Language Processing. 1538–1551

    2024

  38. [45]

    Shuo Yang, Caren Han, Siwen Luo, and Eduard H. Hovy. 2025. MAGIC-VQA: Multimodal And Grounded Inference with Commonsense Knowledge for Visual Question Answering. InFindings of the Association for Computational Linguistics. 16967–16986

    2025

  39. [46]

    Jin Xu, Zhifang Guo, Hangrui Hu, Yunfei Chu, Xiong Wang, Jinzheng He, Yuxuan Wang, Xian Shi, Ting He, Xinfa Zhu, et al. 2025. Qwen3-omni technical report. arXiv preprint arXiv:2509.17765(2025)

    2025 arXiv

  40. [47]

    Zhengrong Yue, Shaobin Zhuang, Kunchang Li, Yanbo Ding, and Yali Wang. 2025. V-Stylist: Video Stylization via Collaboration and Reflection of MLLM Agents. In IEEE/CVF Conference on Computer Vision and Pattern Recognition. 3195–3205

    2025

  41. [49]

    Xinlei Yu, Zhangquan Chen, Yudong Zhang, Shilin Lu, Ruolin Shen, Jiangning Zhang, Xiaobin Hu, Yanwei Fu, and Shuicheng Yan. 2025. Visual Document Un- derstanding and Question Answering: A Multi-Agent Collaboration Framework with Test-Time Scaling.CoRRabs/2508.03404 (2025)

    2025

  42. [50]

    Taolin Zhang, Dongyang Li, Qizhou Chen, Chengyu Wang, and Xiaofeng He. 2025. BELLE: A Bi-Level Multi-Agent Reasoning Framework for Multi-Hop Question Answering. InProceedings of the Association for Computational Linguistics. 4184– 4202

    2025

  43. [51]

    Tao Zhang, Ziqi Zhang, Zongyang Ma, Yuxin Chen, Zhongang Qi, Chunfeng Yuan, Bing Li, Junfu Pu, Yuxuan Zhao, Zehua Xie, Jin Ma, Ying Shan, and Weiming Hu. 2024. mR2AG: Multimodal Retrieval-Reflection-Augmented Generation for Knowledge-Based VQA.CoRRabs/2411.15041 (2024)

    2024 arXiv

  44. [52]

    Shiyue Zhang, Zheng Chong, Xujie Zhang, Hanhui Li, Yuhao Cheng, Yiqiang Yan, and Xiaodan Liang. 2024. GarmentAligner: Text-to-Garment Generation via Retrieval-Augmented Multi-level Corrections. InEuropean Conference on Computer Vision (Lecture Notes in Computer Science, Vol. 1...

    2024

  45. [53]

    RS” denotes random seed, and “#40

    Xin Zou, Yizhou Wang, Yibo Yan, Sirui Huang, Kening Zheng, Junkai Chen, Chang Tang, and Xuming Hu. 2024. Look Twice Before You Answer: Memory-Space Visual Retracing for Hallucination Mitigation in Multimodal Large Language Models.CoRRabs/2410.03577 (2024). M3Prune: Hierarchica...

    2024 arXiv

  46. [55]

    Kunlun Zhu, Hongyi Du, Zhaochen Hong, Xiaocheng Yang, Shuyi Guo, Zhe Wang, Zhenhailong Wang, Cheng Qian, Robert Tang, Heng Ji, and Jiaxuan You

  47. [56]

    InProceedings of the Association for Computational Linguistics

    MultiAgentBench : Evaluating the Collaboration and Competition of LLM agents. InProceedings of the Association for Computational Linguistics. 8580–8622

  48. [2023]

    InIEEE/CVF International Conference on Computer Vision

    Decouple Before Interact: Multi-Modal Prompt Learning for Continual Visual Question Answering. InIEEE/CVF International Conference on Computer Vision. 2941–2950

  49. [2025]

    In Proceedings of the Association for Computational Linguistics

    Agent-RewardBench: Towards a Unified Benchmark for Reward Modeling across Perception, Planning, and Safety in Real-World Multimodal Agents. In Proceedings of the Association for Computational Linguistics. 17521–17541

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.