Pith. sign in

REVIEW 4 major objections 5 minor 174 references

By explicitly planning what to retrieve and where to search before retrieving, MM-R2 outperforms prior multimodal RAG baselines on InfoSeek and Encyclopedic VQA and exposes its retrieval reasoning for inspection.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-02 10:14 UTC pith:DOTXF6VW

load-bearing objection Strong SOTA numbers and a clean system story, but the training pipeline leaks the gold Wikipedia page into the teacher trajectories, so the 'image-grounded, verifiable' claim doesn't stand as written. the 4 major comments →

arxiv 2607.22643 v1 pith:DOTXF6VW submitted 2026-06-24 cs.AI cs.CV

Reason Before You Retrieve: Agentic Planning for Multi-modal RAG

Tianyu Yang , Shir Simon , Zhenzhen Li , Minhao Cheng , Xiangliang Zhang This is my paper
classification cs.AI cs.CV
keywords multimodal retrieval-augmented generationretrieval planningintent groundingknowledge mapagentic retrievaltrajectory datasetreinforcement learningvisual question answering
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper claims that multimodal retrieval-augmented generation performs better when it plans before it searches. MM-R2 first turns an image-question pair into a structured intent triplet—information need, visual referent, and constraints—then routes retrieval through a pre-clustered KnowledgeMap, selecting a semantic unit before issuing a query inside it. The authors construct a 900K-trajectory dataset by having a teacher synthesize multi-step retrieval traces and train a 7B multimodal language model with supervised fine-tuning and reinforcement learning. On InfoSeek, MM-R2 reaches 54.3% overall accuracy, 7.8 points ahead of the strongest prior baseline, and its process-level metrics improve sharply: route recall jumps from 0.23 to 0.74, and fidelity—the degree to which the final answer is reproducible from the exposed evidence—reaches 0.90. If correct, this demonstrates that explicit retrieval planning can improve both accuracy and auditability.

Core claim

The central claim is that decomposing multimodal retrieval into intent grounding, corpus routing, and within-unit retrieval outperforms a single flat similarity search. The Intent-Referent Binding Module outputs a triplet describing the information need, the image-grounded referent, and retrieval constraints; the KnowledgeMap partitions a Wikipedia-derived corpus into ten coarse semantic units; and an agent trained on multi-step trajectories selects a unit, issues a grounded query, summarizes returned evidence, and decides whether to continue. Experiments report 54.1%, 56.0%, and 54.3% on InfoSeek Unseen-Q, Unseen-E, and All, improving over the strongest prior dense-retriever baselines by 6.

What carries the argument

The load-bearing mechanism is a two-part decomposition. The Intent-Referent Binding Module turns the multimodal query into a structured intent triplet, making 'what to retrieve' explicit before any search. The KnowledgeMap partitions the corpus into ten semantically coherent retrieval units, so the agent's first action is a routing decision—choosing 'where to search'—followed by a query generated specifically for that unit. This agent is trained on 900K synthesized trajectories using a two-stage post-training recipe: supervised fine-tuning teaches the structure of multi-step retrieval, and Group Relative Policy Optimization refines the policy under a reward that scores both routing accuracy

Load-bearing premise

The claim of image-grounded retrieval planning assumes that the teacher's synthesized trajectories genuinely reflect visual reasoning about the image, rather than being steered by the gold Wikipedia entry and selected knowledge unit supplied in the teacher prompt.

What would settle it

Run the trained agent on a set of queries where the image is swapped for an unrelated one while the question stays the same; if the chosen retrieval unit, query, and final answer do not change, the image has no causal influence on the retrieval plan, refuting the grounding claim.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Structured routing should reduce retrieval drift and improve precision whenever a question references a specific visual entity, making hierarchical search a viable alternative to flat top-k retrieval on large unstructured corpora.
  • Explicit intent states and exposed retrieval traces provide a built-in audit trail: an answer can be traced to the specific knowledge unit and evidence summary that produced it.
  • The two-stage training recipe shows that a large supervised trajectory dataset carries most of the performance gain, with reinforcement learning adding smaller but consistent refinements.
  • The gains on both InfoSeek and Encyclopedic VQA suggest the approach transfers across benchmarks with different question types and knowledge structures.
  • Process-level metrics such as route recall and fidelity establish a template for evaluating agentic retrieval systems beyond final accuracy.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Because the teacher for trajectory synthesis was given the gold Wikipedia ID, title, summary, and the selected KnowledgeMap unit, and because the intent-binding module was prompted only with an image path string, some of the routing signal may come from benchmark leakage rather than visual understanding; ablating those inputs from the teacher prompt would reveal how much of the gain survives.
  • The KnowledgeMap's ten coarse units are a practical but crude partition; a finer-grained or learned hierarchy could plausibly push accuracy further without changing the training framework.
  • The verifiability metrics introduced here could serve as a blueprint for auditing other retrieval agents, making 'show your work' a standard evaluation axis.
  • If the visual grounding concern is confirmed, the method's advantage may rest on textual priors and dataset regularities rather than true image reasoning, which would limit generalization to genuinely novel visual scenes.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes MM-R2, a multimodal agentic retrieval framework that first extracts a structured intent triplet (information need, grounded referent, constraints) from an image-question pair, then routes retrieval over a clustered KnowledgeMap before issuing within-unit queries. The authors construct a 900K-trajectory dataset (MM-R2-Traj) via a teacher model and train a Qwen2-VL-7B agent with SFT followed by GRPO, with rewards for routing quality and answer correctness. On InfoSeek and Encyclopedic VQA they report SOTA accuracy (54.3% All on InfoSeek, 39.4% on Enc-VQA) and improved process-level metrics (Route Recall@1 0.74, Fidelity 0.90). The paper claims the gains come from explicit image-grounded pre-retrieval reasoning and a structured search space.

Significance. If the claims held, the paper would be a meaningful step toward transparent agentic mRAG: it introduces an explicit routing interface, a large trajectory dataset, and a two-axis evaluation that goes beyond final accuracy. The process-level metrics (route recall, fidelity, LLM-as-a-Judge) are a useful template for the community. However, as detailed below, the training-data construction and the implemented input modalities do not support the claimed image-grounded planning mechanism. The central empirical claim—that SOTA accuracy and verifiable trajectories stem from reasoning about the visual referent—is not established by the current evidence. The external evaluation on official splits is a strength, but it cannot compensate for the fact that the learned policy may simply imitate oracle-informed routes and benchmark priors.

major comments (4)
  1. [Appendix B; Section 4.1] The teacher prompt's 'Known Inputs' include the gold Wikipedia ID, title, summary, the selected KnowledgeMap unit, and the retrieved evidence. Section 4.1's assurance that the reference answer is used only for post-hoc filtering does not address the fact that the gold page's identity is an input to trace generation. SFT (Eq. 6) trains the agent to imitate these oracle-conditioned traces, and the GRPO routing reward (Eqs. 10-11) is assigned according to whether the selected branch is 'appropriate.' Consequently, Route Recall@1=0.74 (Section 6.3) and a large part of the SFT gain in Table 3a may reflect imitation of answer-page-conditioned routing rather than learning to discover the correct unit from the multimodal query. This is load-bearing for the contributions claimed in Sections 1 and 6.2.
  2. [Appendix C/E; Eq. (1)] Eq. (1) defines Φ as mapping (I,Q) to (i,r,c), and Section 3.3 states the agent receives I. The actual prompt templates in Appendices C and E provide only an 'Image Path' string, and Appendix C says 'images as lightweight substitutes for raw image inputs.' Unless the image is loaded and encoded separately—which is not described—the system does not process the image at all. Thus the 'grounded visual referent' component of Eq. (1) is not implemented, and the process-level improvements in Figure 4 cannot be attributed to visual grounding. Appendix B's claim that the teacher uses the image directly also contradicts the template shown. This is an internal inconsistency in a core mechanism.
  3. [Table 3b; Section 6.4] The module ablation has no flat-retrieval trained control. The rows vary intent binding, KnowledgeMap, routing, and agentic loop, but no condition trains a comparable agent on flat dense retrieval over the same corpus with the same trajectory dataset. The largest jump in training ablation (Table 3a) is from base Qwen2-VL to SFT, and Table 3b's first row (22.1) is not a flat-retrieval agent trained on MM-R2-Traj. Without such a control, the 22.1→54.3 cumulative comparison conflates the effect of the routing scaffold with the effect of SFT/RL on a large answer-conditioned trajectory set, and the claim that KnowledgeMap routing and the agentic loop are responsible for the gains is unsupported.
  4. [Section 4.3, Eq. (5)] The routing reward r^route_{i,t} is defined as 'whether the selected branch at step t is appropriate,' but the paper never specifies how appropriateness is computed. If it uses the gold unit—the natural reading given the trajectory synthesis in Appendix B—then the RL reward itself is oracle-conditioned, compounding the SFT leakage. The authors should specify the computation and provide an ablation with λ_route=0 to isolate the contribution of the routing reward.
minor comments (5)
  1. [Appendix overview] The appendix overview contains an unresolved cross-reference: 'Appendix ??'.
  2. [Appendix C] The sentence 'During large-scale preprocessing, we use images as lightweight substitutes for raw image inputs' is confusing; it likely should read 'image paths.' As written, it obscures whether the actual image is used.
  3. [Eq. (7)] The clipping uses ε_low and ε_high without defining their values; standard GRPO uses a single symmetric ε. Please define them.
  4. [Figure 4] The baseline 'Qwen2-VL' should be described in the process-level comparison: does it perform flat retrieval, and how is its Route Recall defined when it has no routing mechanism?
  5. [Table 2] Please clarify the relationship between Top-1/Top-2/Top-5 and the maximum retrieval budget, and specify why several baseline entries are missing.

Circularity Check

1 steps flagged

Oracle-conditioned teacher trajectories make the routing/fidelity 'predictions' partially fitted labels; final answer accuracy remains externally evaluated.

specific steps
  1. fitted input called prediction [Appendix B, Prompt Template for Trajectory Synthesis; used by Section 4.1 SFT and Section 6.3 process metrics]
    "Known Inputs: ... Selected KnowledgeMap Unit:{selected unit} Wikipedia ID:{wiki id} Wikipedia Title:{wiki title} Wikipedia Summary:{wiki summary} Retrieved Evidence:{retrieved evidence} Instruction: Generate a structured retrieval trajectory that reflects how an agent should solve the question through multi-step retrieval."

    The trajectory dataset is the SFT supervision (Eq. 6) for routing and evidence use. The teacher is handed the selected retrieval unit and the gold Wikipedia page before generating the <tool call> and <tool response>; it is a formatter of answer-location inputs, not a planner discovering where to search from (I,Q). The paper's Section 4.1 disclaimer that the reference answer is 'used only for post-hoc filtering' is undercut by these known inputs. Consequently, Route Recall@1 (0.74) and Fidelity (0.90) in Section 6.3 measure imitation of route labels that entered the synthesis prompt, not independent inference of the correct unit from the image-question pair. This is a fitted-input-called-prediction pattern for the process-level claims.

full rationale

The final accuracy claims (InfoSeek 54.1/56.0/54.3; Enc-VQA 39.4) are evaluated on official held-out splits and are not outputs of the trajectory synthesizer, so they are not formally circular. However, the paper's joint claim of 'interpretable and verifiable retrieval trajectories' is supported by process metrics that are trained on answer-conditioned traces: Appendix B supplies the teacher with the gold unit, Wikipedia ID/title/summary, and retrieved evidence, and the SFT/GRPO pipeline then fits the agent to imitate those traces. The later Route Recall@1 and Fidelity results therefore partly recapitulate the labels built into the supervision. Additionally, Appendix C's and Appendix E's prompts receive only an 'Image Path' string, not image content, so the claimed visual grounding of the referent in Eq. (1) is not implemented as written; this is a correctness limitation that compounds the process-level circularity but does not by itself make the held-out accuracy circular. No load-bearing self-citation chain is present. Score 5 reflects a partial circularity confined to the process-level/interpretability claims, with the external benchmark numbers retaining independent value.

Axiom & Free-Parameter Ledger

5 free parameters · 5 axioms · 0 invented entities

The central empirical claims rest on a large synthetic trajectory corpus whose generation uses the gold Wikipedia page and only an image path, plus a hand-chosen 10-unit KnowledgeMap and manually set reward weights. The number of free design choices is high, and no code or data is released.

free parameters (5)
  • Number of KnowledgeMap units = 10
    Chosen via automated HDBSCAN configuration search; routing granularity directly affects routing difficulty and the Route Recall metric.
  • HDBSCAN configuration = unspecified
    Selected under 'semantic coherence, cluster balance, assignment coverage, downstream retrieval utility'; exact hyperparameters not given.
  • Reward weights λroute, λans = 0.1, 1.0
    Set by hand for the GRPO reward; the routing component is weighted ten times lower than the answer component.
  • Chunk/truncation sizes for indexing = 800/100/1600 chars
    Passage chunking choices in Appendix D that affect retrieval granularity.
  • Maximum retrieval budget = not stated
    The multi-step loop termination condition is never specified.
axioms (5)
  • domain assumption The gold answers for InfoSeek and Enc-VQA lie inside the InfoSeek-derived Wikipedia corpus
    The entire RAG evaluation presumes the corpus contains the answer passages; used implicitly in Sections 3.2 and 5.
  • domain assumption An approximate partition of the corpus into 10 coarse topic clusters is a useful routing scaffold
    KnowledgeMap quality is not quantitatively validated against other partitions; asserted in Section 3.2.
  • domain assumption LLM-as-a-Judge (GPT-4o) scores for trace-answer consistency and evidence support are valid proxies for verifiability
    Used in Section 5.1/6.3 without inter-annotator agreement or calibration against human judgments.
  • ad hoc to paper The routing score r^route can be assigned per step according to 'whether the selected branch is appropriate'
    Section 4.3 does not define how appropriateness is computed; if it uses gold-unit membership, the RL signal encodes answer-location knowledge.
  • ad hoc to paper Teacher trajectories synthesized using the gold Wikipedia page and only an image path are valid demonstrations of image-grounded retrieval planning
    This is the load-bearing premise of the whole training pipeline and is not checked; the paper's own example shows the 'grounded' referent is not derivable from the filename.

pith-pipeline@v1.3.0-alltime-deepseek · 125 in / 17940 out tokens · 251508 ms · 2026-08-02T10:14:21.067236+00:00 · methodology

0 comments
read the original abstract

Multimodal retrieval-augmented generation (mRAG) aims to answer image-text queries with external knowledge, but most existing systems still retrieve directly from raw multimodal input over a flat evidence space. This design often struggles with two key challenges: the retrieval target is under-specified because the question intent must be grounded to the correct visual referent, and the search space is weakly structured, forcing semantically distinct evidence to compete in a single global ranking step. We propose MM-R2, a multimodal agentic retrieval framework that reasons before retrieval by explicitly modeling both what to retrieve and where to search. MM-R2 first constructs an intent-grounded retrieval state from the image-question pair, capturing the information need, grounded referent, and retrieval constraints. It then performs retrieval over a structured KnowledgeMap, where the agent selects relevant retrieval units before issuing grounded queries within them. To enable this capability, we build MM-R2-Traj, a large-scale trajectory dataset of multi-step retrieval processes, and adopt a two-stage post-training strategy with supervised fine-tuning and GRPO. Experiments on Infoseek and Encyclopedic VQA datasets show that MM-R2 substantially outperforms strong baselines on answer accuracy while also yielding more interpretable and verifiable retrieval trajectories.

Figures

Figures reproduced from arXiv: 2607.22643 by Minhao Cheng, Shir Simon, Tianyu Yang, Xiangliang Zhang, Zhenzhen Li.

Figure 1
Figure 1. Figure 1: Overview of our MM-R2 pipeline. Given an image-question pair, MM-R2 agent first constructs an intent-grounded retrieval state, then reasons over which KnowledgeMap unit(s) to access, retrieves evidence within the selected unit(s), summarizes the retrieved evidence, and determines whether additional retrieval is needed or whether sufficient evidence has been collected to generate the final answer. query (Lo… view at source ↗
Figure 2
Figure 2. Figure 2: Offline construction of the KnowledgeMap (a): the corpus is pre-structured into semantically coherent retrieval units through embedding, clustering, metadata generation, and quality checking. Synthesis of MM-R2-Traj (b): given an image-question pair and the KnowledgeMap, a teacher model synthesizes structured multi-step retrieval trajectories, including intent states, reasoning, tool calls, tool responses,… view at source ↗
Figure 3
Figure 3. Figure 3: Overview of the two-stage training strategy: SFT+GRPO first predicts a routing action at that selects one or more retrieval units from K, and then generates a grounded retrieval query qt conditioned on (I, Q, z, ht , at). In this way, routing determines where to search, while query generation determines what to search for within the selected region. The action (at , qt) is executed by an external retrieval… view at source ↗
Figure 4
Figure 4. Figure 4: Process-level verifiability on InfoSeek. MM-R2 improves route quality and judge￾based trace quality over the Qwen2-VL baseline, while producing more concise evidence summaries and fidelity values much closer to 1. 6.3 Process-level Effectiveness As shown in [PITH_FULL_IMAGE:figures/full_fig_p010_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Additional qualitative case comparisons. In each example, the comparison method (or the preceding prediction) produces an incorrect answer, while MM-R2 predicts the correct answer through a structured retrieval trajectory with explicit intent grounding, retrieval actions, evidence summarization, and final answer generation. These cases illus￾trate how MM-R2 benefits from explicit retrieval planning and evi… view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

174 extracted references · 3 canonical work pages

  1. [1]

    Langley , title =

    P. Langley , title =. Proceedings of the 17th International Conference on Machine Learning (ICML 2000) , address =. 2000 , pages =

    2000

  2. [2]

    T. M. Mitchell. The Need for Biases in Learning Generalizations. 1980

    1980

  3. [3]

    M. J. Kearns , title =

  4. [4]

    Machine Learning: An Artificial Intelligence Approach, Vol. I. 1983

    1983

  5. [5]

    R. O. Duda and P. E. Hart and D. G. Stork. Pattern Classification. 2000

    2000

  6. [6]

    Suppressed for Anonymity , author=

  7. [7]

    Newell and P

    A. Newell and P. S. Rosenbloom. Mechanisms of Skill Acquisition and the Law of Practice. Cognitive Skills and Their Acquisition. 1981

    1981

  8. [8]

    A. L. Samuel. Some Studies in Machine Learning Using the Game of Checkers. IBM Journal of Research and Development. 1959

    1959

  9. [9]

    KAT : A Knowledge Augmented Transformer for Vision-and-Language

    Gui, Liangke and Wang, Borui and Huang, Qiuyuan and Hauptmann, Alexander and Bisk, Yonatan and Gao, Jianfeng. KAT : A Knowledge Augmented Transformer for Vision-and-Language. Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. 2022. doi:10.18653/v1/2022.naacl-main.70

    doi:10.18653/v1/2022.naacl-main.70 2022

  10. [10]

    Advances in neural information processing systems , volume=

    Revive: Regional visual representation matters in knowledge-based visual question answering , author=. Advances in neural information processing systems , volume=

  11. [11]

    arXiv preprint arXiv:2210.03809 , year=

    Retrieval augmented visual question answering with outside knowledge , author=. arXiv preprint arXiv:2210.03809 , year=

    Pith/arXiv arXiv

  12. [12]

    M u RAG : Multimodal Retrieval-Augmented Generator for Open Question Answering over Images and Text

    Chen, Wenhu and Hu, Hexiang and Chen, Xi and Verga, Pat and Cohen, William. M u RAG : Multimodal Retrieval-Augmented Generator for Open Question Answering over Images and Text. Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing. 2022. doi:10.18653/v1/2022.emnlp-main.375

    work page doi:10.18653/v1/2022.emnlp-main.375 2022

  13. [13]

    Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages=

    Wiki-llava: Hierarchical retrieval-augmented generation for multimodal llms , author=. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages=

  14. [14]

    arXiv preprint arXiv:2407.12735 , year=

    EchoSight: Advancing visual-language models with Wiki knowledge , author=. arXiv preprint arXiv:2407.12735 , year=

    Pith/arXiv arXiv

  15. [15]

    arXiv preprint arXiv:2410.10594 , year=

    Visrag: Vision-based retrieval-augmented generation on multi-modality documents , author=. arXiv preprint arXiv:2410.10594 , year=

    Pith/arXiv arXiv

  16. [16]

    Proceedings of the IEEE/CVF International Conference on Computer Vision , pages=

    M3DocVQA: Multi-modal Multi-page Multi-document Understanding , author=. Proceedings of the IEEE/CVF International Conference on Computer Vision , pages=

  17. [17]

    arXiv preprint arXiv:2411.15041 , year=

    mR2AG: Multimodal Retrieval-Reflection-Augmented Generation for Knowledge-Based VQA , author=. arXiv preprint arXiv:2411.15041 , year=

    Pith/arXiv arXiv

  18. [18]

    arXiv preprint arXiv:2410.08876 , year=

    Rora-vlm: Robust retrieval-augmented vision language models , author=. arXiv preprint arXiv:2410.08876 , year=

    Pith/arXiv arXiv

  19. [19]

    arXiv preprint arXiv:2506.02544 , year=

    CoRe-MMRAG: Cross-Source Knowledge Reconciliation for Multimodal RAG , author=. arXiv preprint arXiv:2506.02544 , year=

    Pith/arXiv arXiv

  20. [20]

    arXiv preprint arXiv:2504.10074 , year=

    MMKB-RAG: A Multi-Modal Knowledge-Based Retrieval-Augmented Generation Framework , author=. arXiv preprint arXiv:2504.10074 , year=

    Pith/arXiv arXiv

  21. [21]

    Yu , booktitle=

    Yangning Li and Yinghui Li and Xinyu Wang and Yong Jiang and Zhen Zhang and Xinran Zheng and Hui Wang and Hai-Tao Zheng and Fei Huang and Jingren Zhou and Philip S. Yu , booktitle=. Benchmarking Multimodal Retrieval Augmented Generation with Dynamic. 2025 , url=

    2025

  22. [22]

    arXiv preprint arXiv:2506.20670 , year=

    MMSearch-R1: Incentivizing LMMs to Search , author=. arXiv preprint arXiv:2506.20670 , year=

    Pith/arXiv arXiv

  23. [23]

    The eleventh international conference on learning representations , year=

    React: Synergizing reasoning and acting in language models , author=. The eleventh international conference on learning representations , year=

  24. [24]

    Advances in Neural Information Processing Systems , volume=

    Toolformer: Language models can teach themselves to use tools , author=. Advances in Neural Information Processing Systems , volume=

  25. [25]

    arXiv preprint arXiv:2303.04671 , year=

    Visual chatgpt: Talking, drawing and editing with visual foundation models , author=. arXiv preprint arXiv:2303.04671 , year=

    Pith/arXiv arXiv

  26. [26]

    arXiv preprint arXiv:2303.11381 , year=

    Mm-react: Prompting chatgpt for multimodal reasoning and action , author=. arXiv preprint arXiv:2303.11381 , year=

    Pith/arXiv arXiv

  27. [27]

    Advances in Neural Information Processing Systems , volume=

    Hugginggpt: Solving ai tasks with chatgpt and its friends in hugging face , author=. Advances in Neural Information Processing Systems , volume=

  28. [28]

    2022 , eprint=

    WebGPT: Browser-assisted question-answering with human feedback , author=. 2022 , eprint=

    2022

  29. [29]

    2022 , eprint=

    MRKL Systems: A modular, neuro-symbolic architecture that combines large language models, external knowledge sources and discrete reasoning , author=. 2022 , eprint=

    2022

  30. [30]

    2023 , eprint=

    Gorilla: Large Language Model Connected with Massive APIs , author=. 2023 , eprint=

    2023

  31. [31]

    2023 , eprint=

    ToolLLM: Facilitating Large Language Models to Master 16000+ Real-world APIs , author=. 2023 , eprint=

    2023

  32. [32]

    2023 , eprint=

    ViperGPT: Visual Inference via Python Execution for Reasoning , author=. 2023 , eprint=

    2023

  33. [33]

    2023 , eprint=

    LLaVA-Interactive: An All-in-One Demo for Image Chat, Segmentation, Generation and Editing , author=. 2023 , eprint=

    2023

  34. [34]

    2024 , eprint=

    GPT-4V(ision) is a Generalist Web Agent, if Grounded , author=. 2024 , eprint=

    2024

  35. [35]

    2023 , eprint=

    Measuring and Narrowing the Compositionality Gap in Language Models , author=. 2023 , eprint=

    2023

  36. [36]

    arXiv preprint arXiv:2302.11713 , year=

    Can pre-trained vision and language models answer visual information-seeking questions? , author=. arXiv preprint arXiv:2302.11713 , year=

    Pith/arXiv arXiv

  37. [37]

    ACL 2019-57th Annual Meeting of the Association for Computational Linguistics , year=

    What does BERT learn about the structure of language? , author=. ACL 2019-57th Annual Meeting of the Association for Computational Linguistics , year=

    2019

  38. [38]

    , author=

    hdbscan: Hierarchical density based clustering. , author=. J. Open Source Softw. , volume=

  39. [39]

    ADDMU : Detection of Far-Boundary Adversarial Examples with Data and Model Uncertainty Estimation

    Yin, Fan and Li, Yao and Hsieh, Cho-Jui and Chang, Kai-Wei. ADDMU : Detection of Far-Boundary Adversarial Examples with Data and Model Uncertainty Estimation. Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing. 2022. doi:10.18653/v1/2022.emnlp-main.440

    work page doi:10.18653/v1/2022.emnlp-main.440 2022

  40. [40]

    Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV) , month =

    Yao Li and Min, Martin Renqiang and Lee, Thomas and Yu, Wenchao and Kruus, Erik and Wang, Wei and Hsieh, Cho-Jui , title =. Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV) , month =. 2021 , pages =

    2021

  41. [41]

    Yao Li and Minhao Cheng and Cho-Jui Hsieh and Thomas C. M. Lee , title =. The American Statistician , volume =. 2021 , publisher =. doi:10.1080/00031305.2021.2006781 , URL =

    Pith/arXiv arXiv 2021

  42. [42]

    Trusted Aggregation (

    Joseph Lavond and Minhao Cheng and Yao Li , journal=. Trusted Aggregation (. 2024 , url=

    2024

  43. [43]

    2024 , url=

    FedDecay: Adapting to Data Heterogeneity in Federated Learning With Gradient Decay , author=. 2024 , url=

    2024

  44. [44]

    European Conference on Computer Vision (ECCV) , year=

    AdaDiff: Accelerating Diffusion Models through Step-Wise Adaptive Computation , author=. European Conference on Computer Vision (ECCV) , year=

  45. [45]

    Xuanqing Liu and Yao Li and Chongruo Wu and Cho-Jui Hsieh , booktitle=. Adv-. 2019 , opturl=

    2019

  46. [46]

    Advances in neural information processing systems , volume=

    Scalable demand-aware recommendation , author=. Advances in neural information processing systems , volume=

  47. [47]

    Advances in Neural Information Processing Systems , pages=

    Learning from group comparisons: exploiting higher order interactions , author=. Advances in Neural Information Processing Systems , pages=

  48. [48]

    Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)(acceptance ratio 25.8\ month =

    Tang, Shengkun and Wang, Yaqing and Kong, Zhenglun and Zhang, Tianchi and Yao Li and Ding, Caiwen and Wang, Yanzhi and Liang, Yi and Xu, Dongkuan , title =. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)(acceptance ratio 25.8\ month =. 2023 , pages =

    2023

  49. [49]

    Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (

    Accelerating Dataset Distillation via Model Augmentation , author=. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (. 2023 , pages =

    2023

  50. [50]

    Yao Li and Tang, Tongyi and Hsieh, Cho-Jui and Lee, Thomas C. M. , journal=. Adversarial Examples Detection With Bayesian Neural Network , year=

  51. [51]

    Thirty-eighth Conference on Neural Information Processing Systems (NeurIPS) , year=

    Improving Logits-based Detector without Logits from Black-box LLMs , author=. Thirty-eighth Conference on Neural Information Processing Systems (NeurIPS) , year=

  52. [52]

    Technometrics (Impact Factor 2.988) , pages=

    Uncertainty Quantification for High Dimensional Sparse Nonparametric Additive Models , author=. Technometrics (Impact Factor 2.988) , pages=. 2019 , publisher=

    2019

  53. [53]

    Transactions on Machine Learning Research , issn=

    Biased Dueling Bandits with Stochastic Delayed Feedback , author=. Transactions on Machine Learning Research , issn=. 2024 , url=

    2024

  54. [54]

    arXiv preprint arXiv:2407.04179 , year=

    Defense Against Syntactic Textual Backdoor Attacks with Token Substitution , author=. arXiv preprint arXiv:2407.04179 , year=

    Pith/arXiv arXiv

  55. [55]

    arXiv preprint arXiv:2310.03693 , year=

    Fine-tuning aligned language models compromises safety, even when users do not intend to! , author=. arXiv preprint arXiv:2310.03693 , year=

    Pith/arXiv arXiv

  56. [56]

    arXiv preprint arXiv:2410.09838 , year=

    Uncovering, Explaining, and Mitigating the Superficial Safety of Backdoor Defense , author=. arXiv preprint arXiv:2410.09838 , year=

    Pith/arXiv arXiv

  57. [57]

    arXiv preprint arXiv:2405.16134 , year=

    Breaking the False Sense of Security in Backdoor Defense through Re-Activation Attack , author=. arXiv preprint arXiv:2405.16134 , year=

    Pith/arXiv arXiv

  58. [58]

    arXiv preprint arXiv:2406.05946 , year=

    Safety Alignment Should Be Made More Than Just a Few Tokens Deep , author=. arXiv preprint arXiv:2406.05946 , year=

    Pith/arXiv arXiv

  59. [59]

    arXiv preprint arXiv:2410.12777 , year=

    Meta-Unlearning on Diffusion Models: Preventing Relearning Unlearned Concepts , author=. arXiv preprint arXiv:2410.12777 , year=

    Pith/arXiv arXiv

  60. [60]

    arXiv preprint arXiv:2411.18915 , year=

    MATATA: a weak-supervised MAthematical Tool-Assisted reasoning for Tabular Applications , author=. arXiv preprint arXiv:2411.18915 , year=

    Pith/arXiv arXiv

  61. [61]

    arXiv preprint arXiv:2407.18219 , year=

    Recursive introspection: Teaching language model agents how to self-improve , author=. arXiv preprint arXiv:2407.18219 , year=

    Pith/arXiv arXiv

  62. [62]

    arXiv preprint arXiv:2406.14532 , year=

    RL on Incorrect Synthetic Data Scales the Efficiency of LLM Math Reasoning by Eight-Fold , author=. arXiv preprint arXiv:2406.14532 , year=

    Pith/arXiv arXiv

  63. [63]

    arXiv preprint arXiv:2302.13971 , year=

    Llama: Open and efficient foundation language models , author=. arXiv preprint arXiv:2302.13971 , year=

    Pith/arXiv arXiv

  64. [64]

    International Conference on Learning Representations , year=

    An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale , author=. International Conference on Learning Representations , year=

  65. [65]

    arXiv preprint arXiv:2403.08295 , year=

    Gemma: Open models based on gemini research and technology , author=. arXiv preprint arXiv:2403.08295 , year=

    Pith/arXiv arXiv

  66. [66]

    arXiv preprint arXiv:2108.07258 , year=

    On the opportunities and risks of foundation models , author=. arXiv preprint arXiv:2108.07258 , year=

    Pith/arXiv arXiv

  67. [67]

    Journal of machine learning research , volume=

    Exploring the limits of transfer learning with a unified text-to-text transformer , author=. Journal of machine learning research , volume=

  68. [68]

    Advances in neural information processing systems , volume=

    Language models are few-shot learners , author=. Advances in neural information processing systems , volume=

  69. [69]

    Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages=

    Efficient conditional pre-training for transfer learning , author=. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages=

  70. [70]

    Communications of the ACM , volume=

    Green ai , author=. Communications of the ACM , volume=. 2020 , publisher=

    2020

  71. [71]

    Edward J Hu and yelong shen and Phillip Wallis and Zeyuan Allen-Zhu and Yuanzhi Li and Shean Wang and Lu Wang and Weizhu Chen , booktitle=. Lo. 2022 , url=

    2022

  72. [72]

    Advances in Neural Information Processing Systems , volume=

    Towards stable backdoor purification through feature shift tuning , author=. Advances in Neural Information Processing Systems , volume=

  73. [73]

    International Conference on Learning Representations , year=

    Fine-Tuning can Distort Pretrained Features and Underperform Out-of-Distribution , author=. International Conference on Learning Representations , year=

  74. [74]

    arXiv preprint arXiv:2506.09433 , year=

    Mitigating Spurious Correlations in LLMs via Causality-Aware Post-Training , author=. arXiv preprint arXiv:2506.09433 , year=

    Pith/arXiv arXiv

  75. [75]

    IEEE Access , volume=

    Badnets: Evaluating backdooring attacks on deep neural networks , author=. IEEE Access , volume=. 2019 , publisher=

    2019

  76. [76]

    Proceedings of the IEEE/CVF International Conference on Computer Vision , pages=

    Invisible backdoor attack with sample-specific triggers , author=. Proceedings of the IEEE/CVF International Conference on Computer Vision , pages=

  77. [77]

    International Conference on Learning Representations , year=

    Dba: Distributed backdoor attacks against federated learning , author=. International Conference on Learning Representations , year=

  78. [78]

    Nature Machine Intelligence , volume=

    Shortcut learning in deep neural networks , author=. Nature Machine Intelligence , volume=. 2020 , publisher=

    2020

  79. [79]

    Proceedings of the AAAI Conference on Artificial Intelligence , volume=

    Inspecting prediction confidence for detecting black-box backdoor attacks , author=. Proceedings of the AAAI Conference on Artificial Intelligence , volume=

  80. [80]

    Advances in Neural Information Processing Systems , volume=

    Jailbroken: How does llm safety training fail? , author=. Advances in Neural Information Processing Systems , volume=

Showing first 80 references.