Pith. sign in

REVIEW 4 major objections 5 minor 45 references

MoDA: Modulation Adapter for Fine-Grained Visual Grounding in Instructional MLLMs

T0 review · 4 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read MoDA claims that a lightweight adapter, which multiplies aligned visual features by an instruction-conditioned channel mask, improves fine-grained visual grounding and reduces hallucinations in multimodal LLMs, with a 12-point gain on…

desk verdict MoDA's gains are real but the attribution to language-conditioned modulation is undercut by its own instruction-free control, the abstract overclaims experiments absent from the body, and the MMVP grader change muddies the headline delta. read the letter →

arxiv 2506.01850 v2 pith:PJLZFXCK submitted 2025-06-02 cs.CV cs.AIcs.LGcs.MM

Wayner Barrios , Andrés Villa , Juan León Alcázar , SouYoung Jin , Bernard Ghanem This is my paper
classification cs.CVcs.AIcs.LGcs.MM
keywords modulationadaptermultimodallargelanguagemodelsfine-grainedvisualgroundinghallucinationreductioninstruction-guidedchannel-wisemaskquestionansweringLLaVA
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper proposes MoDA, a small adapter that re-weights the channel dimensions of already-aligned visual features according to the text instruction, before the features enter the LLM. The claim is that this multiplicative, instruction-guided modulation gives multimodal LLMs finer control over which visual details matter, improving grounding and reducing hallucinations without architectural changes or extra supervision. If true, it offers a cheap, general upgrade: consistent gains across seven benchmarks on LLaVA-1.5 and LLaVA-MoRE, including a 12.0-point jump on MMVP for LLaVA-1.5. The abstract further claims the same mechanism transfers to Qwen3-VL across twelve benchmarks.

What carries the argument

The central object is the modulation mask $M = \sigma(W \cdot F(T, V_{\mathrm{aligned}})) \in [0,1]^E$, produced by two cross-attention transformer layers in which visual tokens query instruction token embeddings. The mask is applied to the aligned visual feature map by element-wise multiplication along the embedding dimension, so every channel of every visual token is independently scaled by a number between 0 and 1. This is the mechanism that is supposed to give the LLM instruction-dependent control over visual detail; the paper's ablations compare it against an instruction-blind MLP mask, self-attention fusion, and placement at different depths.

What would settle it

Re-train LLaVA-1.5 with the MoDA recipe but replace the cross-attention mask with an equal-parameter instruction-blind mask, and evaluate both MoDA and the original LLaVA-1.5 baseline on MMVP using the same Gemma 3 grader; if the +12.0 point gap persists, the claim that instruction guidance drives the gain is falsified.

Watch

Extended reading notes

Core claim

MoDA's central claim is that the entangled, patch-level visual representations produced by CLIP-style encoders are a fixable bottleneck for fine-grained grounding: re-weighting the embedding channels of aligned visual features according to the language instruction lets the LLM focus on task-relevant details. The module computes $\tilde{V}_{\mathrm{aligned}} = V_{\mathrm{aligned}} \odot \sigma(W \cdot F(T, V_{\mathrm{aligned}}))$, where $F$ is a stack of cross-attention transformer layers taking visual tokens as queries and instruction token embeddings as memory, and $\sigma$ clamps the result into a channel-wise mask in $[0,1]^E$. Placed after the existing adapter and before the LLM, this mask scales each dimension of each visual token. The paper reports that MoDA improves all seven benchmarks it evaluates against the LLaVA-1.5 and LLaVA-MoRE baselines, with the largest gain on MMVP (from 24.0 to 36.0 for LLaVA-1.5) and further gains when the visual encoder is upgraded to SigLIP-S2.

Load-bearing premise

The load-bearing premise is that the instruction-conditioned channel mask, not the added trainable parameters, retraining, or a different MMVP grader, is what produces the reported gains.

Editorial extensions

If this is right

  • If MoDA is correct, adding it to LLaVA-1.5 raises MMVP accuracy from 24.0 to 36.0, showing that channel-wise modulation can reduce the fine-grained hallucinations that patch-entangled CLIP features cause.
  • Because MoDA improves POPE, GQA, ScienceQA, LLaVA-Wild, and MMBench for LLaVA-1.5 and also lifts LLaVA-MoRE with both CLIP and SigLIP-S2 encoders, the benefit is not tied to a single architecture or encoder.
  • Since MoDA adds under 1% FLOPs, needs no extra supervision, and attaches during the standard instruction-tuning stage, it can be dropped into existing two-stage pipelines with minimal cost.
  • The gains grow with a stronger visual encoder, such as ScienceQA improving from 77.1 to 81.9 and MMVP from 39.3 to 42.7 with SigLIP-S2, so modulation and encoder quality appear to complement each other.
  • If the abstract's Qwen3-VL and 12-benchmark results hold, the same mechanism transfers beyond CLIP-based encoders to newer MLLM families.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • (Editorial inference) The paper's own ablation shows an instruction-blind two-layer MLP mask averaging 64.2 versus 64.3 for the cross-attention MoDA on POPE/GQA/SQA/MMVP, so much of the benefit may come from added trainable capacity or retraining rather than from language conditioning; the headline +12.0 MMVP gain on the Vicuna-7B row has no matched-capacity instruction-free control.
  • (Editorial inference) Because Appendix A reports MMVP scores are graded with Gemma 3 rather than GPT-4, the reported deltas may partly reflect grader choice unless baselines are re-evaluated with the same grader.
  • (Editorial inference) The full-text experiments tabulate seven benchmarks and two architecture families, while the abstract claims generalization to Qwen3-VL and twelve benchmarks; those broader claims do not appear in the provided tables and await the missing experimental support.
  • (Editorial inference) Since MoDA operates in the aligned embedding space and only needs the text instruction, it could in principle be attached to other two-stage vision-language models, or extended to video or audio inputs where a language query is available.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. This paper proposes MoDA, a lightweight adapter inserted after the visual-language alignment stage of an MLLM. MoDA generates a channel-wise multiplicative mask from instruction tokens via cross-attention over aligned visual features (Eq. 2), then applies that mask before the LLM decodes. The authors integrate MoDA into LLaVA-1.5 and LLaVA-MoRE, compare against prior systems on seven benchmarks, and report gains including +12.0 points on MMVP for LLaVA-1.5. Ablations cover MoDA architecture, auxiliary loss, language backbone, and vision encoder, with additional depth and placement studies in the appendix.

Significance. If the reported gains hold, MoDA would be a useful and inexpensive plug-in for improving fine-grained visual grounding and reducing hallucination, with under 1% FLOP overhead. The paper has clear strengths: open code is promised, the architecture is simple, and the evaluation spans multiple backbones and benchmarks. However, the current evidence does not isolate the proposed language-conditioned modulation mechanism from added trainable capacity and evaluation-pipeline changes, so the significance of the central mechanistic claim is not yet established.

major comments (4)
  1. [§4.2, Table 2] The ablation evidence does not support the paper's central claim that instruction-conditioned cross-attention modulation causes the reported gains. In Table 2, the instruction-blind Linear (MLP) variant (row 5) averages 64.2 on POPE/GQA/SQA/MMVP with MMVP 28.1, while the Cross-Attention variant (row 7) averages 64.3 with MMVP 28.7; that 0.1-point average gap is within single-run noise, and no seeds or error bars are reported. The headline +12.0 MMVP gain on the Vicuna-7B row has no instruction-free or matched-capacity control. Without such a control, the gains could equally be attributed to added trainable capacity or training-pipeline differences rather than to the language-conditioned mask defined in Eq. (2).
  2. [§4.2, Table 1, Appendix A] The MMVP column is regraded with Gemma 3 instead of GPT-4, yet the table does not state which entries were reproduced with this grader and which were taken from published work. Because the reported +12.0 MMVP delta for LLaVA-1.5 depends on a direct comparison with previously published baseline numbers, the grader mismatch is a confound. The authors should re-grade all baselines and MoDA variants with the same grader and report both sets of numbers, or explicitly qualify that the delta is not comparable to the literature values.
  3. [Abstract vs. Sections 1 and 4] The abstract claims evaluation across 12 benchmarks and generalization to Qwen3-VL, but the experimental sections report only seven benchmarks and only LLaVA-1.5 and LLaVA-MoRE. The Qwen3-VL results and at least five of the named benchmarks are not present in the main text or any table. The phrase 'consistent gains across all three families' is also contradicted at the column level: in Table 1, the LLaVA-MoRE CLIP row drops on MMBench-Cn (68.2 to 66.1), and the SigLIP-S2 row drops on LLaVA-Wild (72.0 to 67.6) and MMBench-Cn (68.0 to 63.6). These claims need to be either supported by the reported experiments or removed.
  4. [§4.2 text comparisons] Several comparisons in the text ignore the table's own warning that models pretrained under substantially different data distributions are not directly comparable. For example, the claim that LLaVA-1.5+MoDA 'outperforms state-of-the-art MLLM' using a +21.5 GQA margin over BLIP-2 compares a retrained MoDA model with a published baseline under different data and training setups. Since MoDA rows are retrained and evaluated with a different MMVP grader, the deltas conflate pipeline changes with the adapter's contribution; the paper should report matched, properly controlled comparisons or soften these claims.
minor comments (5)
  1. [Eq. (3)] There is a typo in the sentence introducing Eq. (3): 'reamains' should be 'remains'.
  2. [§4.3.2 and Table 2] The text states that the Vicuna-7B configuration achieves an average score of 64.0, but the corresponding row in Table 2 (Cross-Attention, no loss, Vicuna-7B) averages (87.1+62.5+71.0+36.0)/4 = 64.15, which rounds to 64.2; please reconcile the number.
  3. [Throughout] The spelling of the baseline name is inconsistent: 'LLaVA-MoRE' appears in the tables and references, while 'LLaVA-More' and 'LLaV A-More' appear in several places; please standardize.
  4. [§3.1 and §4.1] Section 3.1 says the modulation function is implemented with 'a stack of Transformer Layers', while Section 4.1 specifies two stacked cross-attention layers; please clarify the intended description.
  5. [Appendix B, Table S2] The caption contains 'Not that the first row mirrors line 5 of Table 2', which appears to be a typo for 'Note that'; please fix.

Circularity Check

0 steps flagged · score 1.0 of 10

No circular derivation: MoDA is an empirical adapter evaluated against external benchmarks; the attribution and baseline-comparison concerns are methodological, not circular.

full rationale

MoDA is an empirical adapter module. Equation (2) defines the modulation mask as a learned function of Valigned and T; it is not defined in terms of any benchmark score. The reported MMVP, GQA, ScienceQA, MMBench, and POPE numbers come from held-out external benchmark evaluations, and no benchmark value is used to fit MoDA's weights. There is no fitted parameter renamed as a prediction. Reference [21] is a same-first-author prior work on attention masks, but it is cited only as related work and as an overhead contrast; the paper's central mechanism and training recipe are not justified by [21]. No uniqueness theorem, ansatz, or known result is imported from the authors' own prior work to force the design. Table 2 does show that an instruction-blind MLP matches the cross-attention MoDA (64.2 vs 64.3 average), and the +12.0 MMVP row lacks a matched capacity control; that weakens the causal attribution of gains to instruction-conditioned modulation, but it is a correctness or confound issue rather than a circular definition or fitted-input prediction. The use of published baseline numbers and Gemma 3 as the MMVP grader are evaluation-validity concerns, not circularity. Therefore the paper is empirically self-contained and scores at the no-significant-circularity level.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

The central claims rest on three assumptions: patch-level visual representations are the bottleneck; multiplicative channel gating steered by language is safe and useful; and the training/evaluation pipeline matches the published baselines compared against. The paper gives no causal evidence for the first, its own ablation weakens the second, and the third is threatened by the Gemma 3 grader swap and the use of published baseline numbers. The design choices (depth 2, single-block placement, cross-attention, no L1 loss) were selected by reading the same benchmarks reported as results, which adds a mild selection-on-results risk rather than a circular-derivation risk. MoDA is a standard neural module (cross-attention stack plus sigmoid projection), not a new physical entity, so the invented-entities list is empty.

free parameters (4)
  • MoDA architecture (cross-attention, sigmoid gate, no L1 loss) = 2-layer cross-attention, 16 heads, sigmoid mask, no auxiliary loss
    Selected from Tables 2 and S2-S4 using the same benchmarks reported as headline results; deeper or regularized variants perform worse.
  • MoDA placement = single block at the beginning of the LLM
    Chosen from Table S3 because it beat full-depth placement (69.4 vs 67.5 average) on the reported benchmarks.
  • MMVP grader = Gemma 3 via Ollama (not GPT-4)
    Hand choice reported in the Table 1 footnote and Appendix A; changes the MMVP evaluation protocol relative to published baselines and may shift all MMVP numbers.
  • Training hyperparameters = LR 1e-3/2e-5, batch 256/128, 1 epoch, DeepSpeed stages 2/3
    Copied from LLaVA-1.5 (Table S1). Standard choices, but the results depend on them and they are hand-chosen rather than derived.
assumptions (4)
  • domain assumption Patch-level CLIP representations are the operative bottleneck for fine-grained grounding and hallucination.
    Motivation in Section 1 and the abstract; no causal evidence is provided that patch entanglement causes the benchmark failures, and Section 1 later hedges the claim.
  • domain assumption Channel-wise multiplicative modulation of already-aligned features improves, and does not corrupt, the visual representation passed to the LLM.
    Core design premise of Eq. 1-2 in Section 3.1; undercut by Table 2, where an instruction-blind MLP matches cross-attention performance, and by Table S2, where a 4-layer variant collapses the scores.
  • domain assumption Published baseline scores are comparable to the authors' retrained MoDA models under the same evaluation pipeline.
    Baseline rows in Tables 1-2 are taken from prior publications; MoDA rows are trained by the authors, and MMVP was graded with Gemma 3 rather than GPT-4, so pipeline differences are confounds.
  • domain assumption Single-run benchmark scores are stable enough to support deltas of 0.1 to 12 points.
    No seeds or error bars are reported anywhere; MMVP has only 150 items, making a 12-point delta a small-sample event.

how reviews work

0 comments
Cite this review

Pith. "Pith review of MoDA: Modulation Adapter for Fine-Grained Visual Grounding in Instructional MLLMs." pith.science (2026). https://pith.science/paper/PJLZFXCK

@misc{pith2026250601850,
  author       = {Pith},
  title        = {Pith review of: MoDA: Modulation Adapter for Fine-Grained Visual Grounding in Instructional MLLMs},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/PJLZFXCK}},
  note         = {Machine review of arXiv:2506.01850}
}
read the original abstract

Multimodal Large Language Models (MLLMs) have achieved remarkable success in instruction-following tasks by integrating pretrained visual encoders with large language models (LLMs). However, existing approaches often struggle with fine-grained visual grounding due to semantic entanglement in visual patch representations, where individual patches blend multiple distinct visual elements, making it difficult for models to focus on instruction-relevant details. To address this challenge, we propose MoDA (Modulation Adapter), a lightweight module that enhances visual grounding through instruction-guided channel-wise modulation. Unlike token-level methods such as Q-Former that perform additive feature selection, MoDA operates at the channel level through multiplicative modulation on already-aligned features, enabling fine-grained control over which embedding dimensions are relevant for each instruction. Following the standard LLaVA training protocol, MoDA applies cross-attention between language instructions and pre-aligned visual features, generating dynamic modulation masks without architectural modifications or additional supervision. We evaluate MoDA across 12 benchmarks spanning visual question answering, vision-centric reasoning, and hallucination detection, including recent 2024 benchmarks (MMVP, CV-Bench, MMStar, RealWorldQA), on three distinct MLLM architectures: LLaVA-1.5, LLaVA-MoRE (2025), and Qwen3-VL (2025). MoDA delivers consistent gains across all three families, with +12.0 points on MMVP for the LLaVA-1.5 family and +4.8 points on ScienceQA for the LLaVA-MoRE family, and +4.9 ScienceQA, +4.1 RealWorldQA, and +3.8 GQA on Qwen3-VL, confirming that the gains generalize beyond CLIP-based encoders with minimal overhead (<1% FLOPs). Code is available at https://github.com/waybarrios/MoDA.

Figures

Figures reproduced from arXiv: 2506.01850 by the authors.

Figure 1
Figure 1. Overview of ViT patch representation and our proposed Modulation Adapter (MoDA). (a) ViT splits the input image into fixed-size patches, each projected into a high-dimensional embed￾ding. This rigid partitioning often blends semantically distinct elements (e.g., parts of the dog, toy, floor, and bed within a single patch), leading to entangled representations that hinder fine-grained visual understanding. (b) We int… view at source ↗
Figure 2
Figure 2. Overall Architecture. We outline the two-stage training framework. In Stage 1 (Pre￾training), we train the adapter module using a large-scale multimodal dataset to establish robust visual￾language alignment. In Stage 2 (Instruction Tuning), the model is further trained on instruction￾based data, where the adapter is fine-tuned alongside our proposed MODulation Adapter (MoDA) and a pretrained language model. MoDA is … view at source ↗

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

45 extracted references · 20 canonical work pages

  1. [1]

    Mmlu-pro: A more robust and challenging multi-task language understanding benchmark, 2024

    Yubo Wang, Xueguang Ma, Ge Zhang, Yuansheng Ni, Abhranil Chandra, Shiguang Guo, Weiming Ren, Aaran Arulraj, Xuan He, Ziyan Jiang, Tianle Li, Max Ku, Kai Wang, Alex Zhuang, Rongqi Fan, Xiang Yue, and Wenhu Chen. Mmlu-pro: A more robust and challenging multi-task language understanding benchmark, 2024. URL https://arxiv.org/abs/2406. 01574

    work page 2024

  2. [2]

    Chi, Jeff Dean, Jacob Devlin, Adam Roberts, Denny Zhou, Quoc V

    Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tai, William Fedus, Yunxuan Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, Albert Webson, Shixiang Shane Gu, Zhuyun Dai, Mirac Suzgun, Xinyun Chen, Aakanksha Chowdhery, Alex Castro-Ros, Marie Pellat, Kevin Robinson, Dasha Valter, Sharan Narang, Gaurav Mishra, Adams Yu, Vincent Zhao, Yanping H...

    work page 2024

  3. [3]

    Manning, Christopher Ré, Diana Acosta-Navas, Drew A

    Percy Liang, Rishi Bommasani, Tony Lee, Dimitris Tsipras, Dilara Soylu, Michihiro Yasunaga, Yian Zhang, Deepak Narayanan, Yuhuai Wu, Ananya Kumar, Benjamin Newman, Binhang Yuan, Bobby Yan, Ce Zhang, Christian Cosgrove, Christopher D. Manning, Christopher Ré, Diana Acosta-Navas, Drew A. Hudson, Eric Zelikman, Esin Durmus, Faisal Ladhak, Frieda Rong, Hongyu...

    arXiv 2023

  4. [4]

    The llama 3 herd of models, 2024

    Llama Team, AI @ Meta. The llama 3 herd of models, 2024. URL https://arxiv.org/ abs/2407.21783

    arXiv 2024

  5. [5]

    Qwen2.5 technical report.arXiv preprint arXiv:2412.15115, 2024

    An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jingren Zhou, Junyang Lin, Kai Dang, Keming Lu, Keqin Bao, Kexin Yang, Le Yu, Mei Li, Mingfeng Xue, Pei Zhang, Qin Zhu, Rui Men, Runji Lin, Tianhao Li, Tingyu X...

    arXiv 2024

  6. [6]

    Gemma 3 technical report, 2025

    Gemma Team. Gemma 3 technical report, 2025. URL https://arxiv.org/abs/2503. 19786

    2025

  7. [7]

    Behind the magic, merlim: Multi-modal evaluation benchmark for large image-language models, 2024

    Andrés Villa, Juan Carlos León Alcázar, Alvaro Soto, and Bernard Ghanem. Behind the magic, merlim: Multi-modal evaluation benchmark for large image-language models, 2024. URL https://arxiv.org/abs/2312.02219

    arXiv 2024

  8. [8]

    Eyes wide shut? exploring the visual shortcomings of multimodal llms, 2024

    Shengbang Tong, Zhuang Liu, Yuexiang Zhai, Yi Ma, Yann LeCun, and Saining Xie. Eyes wide shut? exploring the visual shortcomings of multimodal llms, 2024. URL https://arxiv. org/abs/2401.06209

    arXiv 2024

Show all 45 references
  1. [9]

    Brave: Broadening the visual encoding of vision-language models

    Ouguzhan Fatih Kar, Alessio Tonioni, Petra Poklukar, Achin Kulshrestha, Amir Zamir, and Federico Tombari. Brave: Broadening the visual encoding of vision-language models. In European Conference on Computer Vision, 2024. URL https://api.semanticscholar. org/CorpusID:269033274

    2024

  2. [10]

    Eagle: Enhanced visual grounding minimizes hallucinations in instructional multi- modal models, 2025

    Andrés Villa, Juan León Alcázar, Motasem Alfarra, Vladimir Araujo, Alvaro Soto, and Bernard Ghanem. Eagle: Enhanced visual grounding minimizes hallucinations in instructional multi- modal models, 2025. URL https://arxiv.org/abs/2501.02699

    2025 arXiv

  3. [11]

    Ross, and Alireza Fathi

    Ziniu Hu, Ahmet Iscen, Chen Sun, Zirui Wang, Kai-Wei Chang, Yizhou Sun, Cordelia Schmid, David A. Ross, and Alireza Fathi. Reveal: Retrieval-augmented visual-language pre-training with multi-source multimodal knowledge memory. 2023 IEEE/CVF Conference on Computer Vision and Pa...

    2023

  4. [12]

    Kosmos-2: Grounding multimodal large language models to the world

    Zhiliang Peng, Wenhui Wang, Li Dong, Yaru Hao, Shaohan Huang, Shuming Ma, and Furu Wei. Kosmos-2: Grounding multimodal large language models to the world. ArXiv, abs/2306.14824,

    arXiv

  5. [13]

    Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy V . V o, Marc Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel HAZIZA, Francisco Massa, Alaaeldin El-Nouby, Mido Assran, Nicolas Ballas, Wojciech Galuba, Russell Howes, Po-Yao Huang, Shang-Wen Li, Ishan Misra, Michael Rab...

    2024

  6. [14]

    Visualizing and understanding patch interactions in vision transformer

    Jie Ma, Yalong Bai, Bineng Zhong, Wei Zhang, Ting Yao, and Tao Mei. Visualizing and understanding patch interactions in vision transformer. IEEE Transactions on Neural Networks and Learning Systems, 35:13671–13680, 2022. URL https://api.semanticscholar.org/ CorpusID:247410956

    2022

  7. [15]

    four secrets

    Tao Zhou, Yuxia Niu, Huiling Lu, Caiyue Peng, Yujie Guo, and Huiyu Zhou. Vision transformer: To discover the “four secrets” of image patches. Information Fusion, 105:102248, 2024. ISSN 1566-2535. doi: https://doi.org/10.1016/j.inffus.2024.102248. URL https://www. sciencedirect...

    2024

  8. [16]

    Visualization comparison of vision transformers and convolutional neural networks

    Rui Shi, Tianxing Li, Liguo Zhang, and Yasushi Yamaguchi. Visualization comparison of vision transformers and convolutional neural networks. IEEE Transactions on Multimedia, 26: 2327–2339, 2024. doi: 10.1109/TMM.2023.3294805

    2024

  9. [17]

    Mask attention networks: Rethinking and strengthen transformer

    Zhihao Fan, Yeyun Gong, Dayiheng Liu, Zhongyu Wei, Siyuan Wang, Jian Jiao, Nan Duan, Ruofei Zhang, and Xuanjing Huang. Mask attention networks: Rethinking and strengthen transformer. In Kristina Toutanova, Anna Rumshisky, Luke Zettlemoyer, Dilek Hakkani-Tur, Iz Beltagy, Steven...

    2021

  10. [18]

    Multiway dynamic mask attention networks for natural language inference

    Jingfan Tang, Xinqiang Wu, Min Zhang, Xiujie Zhang, and Ming Jiang. Multiway dynamic mask attention networks for natural language inference. J. Comp. Methods in Sci. and Eng., 21(1):151–162, jan 2021. ISSN 1472-7978. doi: 10.3233/JCM-204451. URL https://doi. org/10.3233/JCM-204451

    2021 doi

  11. [19]

    An adaptive masked attention mechanism to act on the local text in a global context for aspect-based sentiment analysis

    Te Lin and Inwhee Joe. An adaptive masked attention mechanism to act on the local text in a global context for aspect-based sentiment analysis. IEEE Access, 11:43055–43066, 2023. doi: 10.1109/ACCESS.2023.3270927

    2023

  12. [20]

    What does self- attention learn from masked language modelling?, 2024

    Riccardo Rende, Federica Gerace, Alessandro Laio, and Sebastian Goldt. What does self- attention learn from masked language modelling?, 2024

    2024

  13. [21]

    Multi-layer learnable attention mask for multimodal tasks,

    Wayner Barrios and SouYoung Jin. Multi-layer learnable attention mask for multimodal tasks,

  14. [22]

    Improved baselines with visual instruction tuning

    Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning. In 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 26286–26296, 2024. doi: 10.1109/CVPR52733.2024.02484

    2024

  15. [23]

    Llava-more: A comparative study of llms and visual backbones for enhanced visual instruction tuning, 2025

    Federico Cocchi, Nicholas Moratelli, Davide Caffagni, Sara Sarto, Lorenzo Baraldi, Marcella Cornia, and Rita Cucchiara. Llava-more: A comparative study of llms and visual backbones for enhanced visual instruction tuning, 2025. URL https://arxiv.org/abs/2503.15621

    2025 arXiv

  16. [24]

    Visual instruction tuning

    Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. In Thirty-seventh Conference on Neural Information Processing Systems, 2023. 11

    2023

  17. [25]

    Sharegpt4v: Improving large multi-modal models with better captions

    Lin Chen, Jinsong Li, Xiaoyi Dong, Pan Zhang, Conghui He, Jiaqi Wang, Feng Zhao, and Dahua Lin. Sharegpt4v: Improving large multi-modal models with better captions. In Computer Vision – ECCV 2024: 18th European Conference, Milan, Italy, September 29–October 4, 2024, Proceeding...

    2024 doi

  18. [26]

    Junnan Li, Dongxu Li, Silvio Savarese, and Steven C. H. Hoi. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. In Interna- tional Conference on Machine Learning, 2023. URL https://api.semanticscholar.org/ CorpusID:256390509

    2023

  19. [27]

    Wenliang Dai, Junnan Li, Dongxu Li, Anthony Meng Huat Tiong, Junqi Zhao, Weisheng Wang, Boyang Albert Li, Pascale Fung, and Steven C. H. Hoi. Instructblip: Towards general- purpose vision-language models with instruction tuning. ArXiv, abs/2305.06500, 2023. URL https://api.sem...

    2023 arXiv

  20. [28]

    Mst: Masked self-supervised transformer for visual representation

    Zhaowen Li, Zhiyang Chen, Fan Yang, Wei Li, Yousong Zhu, Chaoyang Zhao, Rui Deng, Liwei Wu, Rui Zhao, Ming Tang, and Jinqiao Wang. Mst: Masked self-supervised transformer for visual representation. In M. Ranzato, A. Beygelzimer, Y . Dauphin, P.S. Liang, and J. Wortman Vaughan,...

    2021

  21. [29]

    Swinbert: End-to-end transformers with sparse attention for video captioning

    Kevin Lin, Linjie Li, Chung-Ching Lin, Faisal Ahmed, Zhe Gan, Zicheng Liu, Yumao Lu, and Lijuan Wang. Swinbert: End-to-end transformers with sparse attention for video captioning. In CVPR, 2022

    2022

  22. [30]

    Hudson and Christopher D

    Drew A. Hudson and Christopher D. Manning. Gqa: A new dataset for real-world visual reasoning and compositional question answering. In 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 6693–6702, 2019. doi: 10.1109/CVPR.2019. 00686

    2019 doi

  23. [31]

    Shamma, Michael S

    Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin Johnson, Kenji Hata, Joshua Kravitz, Stephanie Chen, Yannis Kalantidis, Li-Jia Li, David A. Shamma, Michael S. Bernstein, and Fei-Fei Li. Visual genome: Connecting language and vision using crowdsourced dense image annotations,

  24. [32]

    Learn to explain: Multimodal reasoning via thought chains for science question answering

    Pan Lu, Swaroop Mishra, Tony Xia, Liang Qiu, Kai-Wei Chang, Song-Chun Zhu, Oyvind Tafjord, Peter Clark, and Ashwin Kalyan. Learn to explain: Multimodal reasoning via thought chains for science question answering. In The 36th Conference on Neural Information Process- ing System...

    2022

  25. [33]

    Evaluating ob- ject hallucination in large vision-language models

    Yifan Li, Yifan Du, Kun Zhou, Jinpeng Wang, Wayne Xin Zhao, and Ji rong Wen. Evaluating ob- ject hallucination in large vision-language models. In Conference on Empirical Methods in Nat- ural Language Processing, 2023. URL https://api.semanticscholar.org/CorpusID: 258740697

    2023

  26. [34]

    Mmbench: Is your multi-modal model an all-around player? In European Conference on Computer Vision, 2023

    Yuanzhan Liu, Haodong Duan, Yuanhan Zhang, Bo Li, Songyang Zhang, Wangbo Zhao, Yike Yuan, Jiaqi Wang, Conghui He, Ziwei Liu, Kai Chen, and Dahua Lin. Mmbench: Is your multi-modal model an all-around player? In European Conference on Computer Vision, 2023. URL https://api.seman...

    2023

  27. [35]

    Qwen-vl: A frontier large vision-language model with versatile abilities

    Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. Qwen-vl: A frontier large vision-language model with versatile abilities. arXiv preprint arXiv:2308.12966, 2023

    2023 arXiv

  28. [36]

    Qwen2.5-vl technical report,

    Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, Humen Zhong, Yuanzhi Zhu, Mingkun Yang, Zhaohai Li, Jianqiang Wan, Pengfei Wang, Wei Ding, Zheren Fu, Yiheng Xu, Jiabo Ye, Xi Zhang, Tianbao Xie, Zesen Cheng, Han...

  29. [37]

    Gonzalez, and Ion Stoica

    Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. Judging LLM-as-a-judge with MT-bench and chatbot arena. In Thirty-seventh Conference on Neural Infor...

    2023

  30. [38]

    Learning transferable visual models from natural language supervision, 2021

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agar- wal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision, 2021. URL https://...

    2021 arXiv

  31. [39]

    Sigmoid loss for language image pre-training, 2023

    Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, and Lucas Beyer. Sigmoid loss for language image pre-training, 2023. URL https://arxiv.org/abs/2303.15343

    2023 arXiv

  32. [40]

    PT” refers to the pre-training stage using large-scale alt-text image–caption data, while “FT

    Baifeng Shi, Ziyang Wu, Maolin Mao, Xin Wang, and Trevor Darrell. When do we not need larger vision models? In NeurIPS 2024 Workshop: Self-Supervised Learning - Theory and Practice, 2024. URL https://openreview.net/forum?id=998NbPvfrD. 13 A Implementation Details Table S1 summ...

    2024

  33. [2016]

    URL https://arxiv.org/abs/1602.07332

    arXiv

  34. [2021]

    doi: 10.18653/v1/2021.naacl-main.135

    Association for Computational Linguistics. doi: 10.18653/v1/2021.naacl-main.135. URL https://aclanthology.org/2021.naacl-main.135

    2021 doi

  35. [2023]

    URL https://api.semanticscholar.org/CorpusID:259262263

  36. [2024]

    URL https://arxiv.org/abs/2406.02761

    arXiv

  37. [2025]

    URL https://arxiv.org/abs/2502.13923. 12

    arXiv

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.