Pith. sign in

REVIEW 4 major objections 5 minor 143 references

Manager: Aggregating Insights from Unimodal Experts in Two-Tower VLMs and MLLMs

T0 review · 4 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A lightweight manager that aggregates multi-layer unimodal representations improves two-tower vision-language models and multimodal LLMs at near-zero added cost.

desk verdict Useful drop-in plugin paper with honest limitations, but the 20-dataset MLLM claim needs the removed datasets and variance estimates before it can be taken at face value. read the letter →

arxiv 2506.11515 v1 pith:2I2GSVDK submitted 2025-06-13 cs.CV cs.CLcs.LG

Xiao Xu , Libo Qin , Wanxiang Che , Min-Yen Kan This is my paper
classification cs.CVcs.CLcs.LG
keywords vision-languagemodelmultimodallargelanguagetwo-towerarchitecturemanagermodulemulti-layerrepresentationaggregationmulti-gridalgorithmzero-shotevaluationvisuallearning
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper proposes a plugin-style module, the manager, that feeds aggregated representations from multiple layers of pre-trained unimodal encoders into a vision-language fusion module, instead of using only last-layer features or fixed layer-by-layer bridges. The claim is that this adaptive aggregation of insights from unimodal experts makes vision-language alignment more comprehensive and cheap: in a two-tower model (ManagerTower) it outperforms METER and BridgeTower under the same 4M-image pre-training and on four downstream tasks, and in the multimodal LLM LLaVA-OV it raises zero-shot averages across 20 datasets by about 1.06 points without multi-grid tiling and 1.44 points with it, at negligible parameter and compute overhead. A sympathetic reader would care because the module is architecture-agnostic, so if the claim holds it is a low-cost way to improve both older two-tower models and modern high-resolution multimodal LLMs.

What carries the argument

The central object is the manager, a lightweight aggregation layer inserted into each cross-modal layer (or, at intervals, into the LLM of an MLLM). Its main form, AAUM, computes per-token aggregation weights $W_A=\mathrm{softmax}(\mathrm{LN}(C^V_{\ell-1})W_M+\epsilon)$ and produces $\tilde{C}^V_\ell=\sum_{i=1}^{6}W_{A,i}\odot\mathrm{LN}(V_{i+6})+W_C\odot\mathrm{LN}(C^V_{\ell-1})$, so each token in each sample can draw differently on the top $N=6$ unimodal layer representations plus the previous cross-modal layer state. In the MLLM setting the paper uses an optimized SAUM variant with zero-initialized weights, no LayerNorm and no softmax, to avoid disturbing the pre-trained LLM early in training. This mechanism decouples the number of unimodal layers exploited from the number of cross-modal layers, and the authors use attention entropy and head-diversity measurements to attribute the gains to more diverse visual attention.

What would settle it

Run the four-baseline comparison on all datasets in the original LLaVA-OV zero-shot suite with no datasets removed and report per-dataset deltas; if the +Grid+Manager advantage over +Grid shrinks to zero or becomes negative once the excluded datasets are included, the 20-dataset averaging claim collapses.

Watch

Extended reading notes

Core claim

On the paper's own terms, the discovery is that layers of a pre-trained unimodal encoder can be treated as experts, and a small trainable manager that learns to combine their outputs—token-by-token in ManagerTower, with simpler static weights in LLaVA-OV-Manager—supplies exactly the level of semantic detail each cross-modal layer needs. This repairs what the authors identify as BridgeTower's flaw: fixed one-to-one bridges force layer-by-layer reuse of unimodal representations and tether the number of bridges to the number of cross-modal layers. With managers, ManagerTower reaches 79.15% on VQAv2 test-std and 86.56% IR@1 on Flickr30K using the same 4M pre-training data as BridgeTower, and LLaVA-OV-Manager improves the zero-shot average on 20 datasets from 50.61% to 51.67% without grids and from 53.87% to 55.21% with grids, with 0.08M extra parameters. The paper further argues that the manager's multi-layer aggregation (depth) and the multi-grid algorithm (width) are orthogonal ways to add visual detail, and that their synergy counters the semantic ambiguity caused by cutting images into grids.

Load-bearing premise

The paper's claim of consistent gains on 20 datasets assumes that the datasets it kept after removing 'few some downstream datasets' are representative of the full evaluation suite, so if the removals systematically dropped cases where managers helped least, the reported averages overstate the general result.

Editorial extensions

If this is right

  • ManagerTower with half the cross-modal layers ($L_C=3$) can beat BridgeTower with $L_C=6$, so the same data and backbones yield better results with less cross-modal depth.
  • Adding managers to LLaVA-OV is cheap: about 0.08M parameters, roughly 2% more FLOPs, and 3–4% more training or inference time, while adding about one absolute point without grids and 1.44 points with grids on the 20-dataset average.
  • The top half of the visual encoder carries most of the benefit; including all layers hurts, so practitioners can select a fixed layer range and reuse it across architectures.
  • The manager and the multi-grid algorithm are complementary: managers help most where grid splitting cuts objects and creates ambiguity, such as instance location and counting on SEED-Bench and artistic or handwriting text recognition on OCRBench.
  • Because the manager works with different visual and textual backbones and with both two-tower and MLLM architectures, the same recipe transfers across model families without retraining the unimodal encoders.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the depth/width decomposition is the right explanation, managers should also stack with other width-style plugins—different tile sizes, high-resolution encoders, or multi-scale crops—rather than only with LLaVA-OV's specific multi-grid scheme; that is testable with the same four-baseline recipe.
  • The 20-dataset average should be re-computed on the full original LLaVA-OV evaluation suite; Appendix C-E says the authors removed 'few some downstream datasets' for efficiency and robustness, and the robustness of the central claim depends on whether those omitted datasets also benefit from the manager.
  • The attention-entropy mechanism suggests managers might help in other settings where vision tokens are ambiguous, such as small objects or heavily cropped inputs, even without grid tiling; a controlled experiment varying input resolution while holding architecture fixed could test this.
  • AAUM's failure to beat SAUM in the causal LLM setting hints that a bidirectional or causally compatible query might unlock more of the manager's potential in MLLMs, a direction the paper itself raises as future work.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes Manager, a lightweight module that aggregates representations from multiple layers of pretrained unimodal encoders and injects them into cross-modal or LLM layers. In the two-tower VLM setting, ManagerTower replaces BridgeTower's one-hot layer-wise bridges with learned static (SAUM) or adaptive (AAUM) managers, reporting gains on VQAv2, SNLI-VE, NLVR2, and Flickr30K under matched pretraining and finetuning. In the MLLM setting, LLaVA-OV-Manager injects visual managers into LLaVA-OneVision, with or without the multi-grid algorithm, and reports zero-shot gains on 20 datasets, together with attention-diversity analyses and case studies.

Significance. If the empirical claims are sound, the Manager plugin is a useful, architecture-agnostic contribution: it is lightweight, compatible with different encoders and cross-modal depths, and the paper provides a fairly extensive set of controlled comparisons, ablations, and computational-budget measurements. The ManagerTower results on four downstream tasks are internally consistent and appear to be a credible extension of BridgeTower. The MLLM extension is ambitious and the depth/width synergy viewpoint with the multi-grid algorithm is interesting. However, the headline 20-dataset claim is currently not auditable because the paper does not disclose which datasets were removed from the original LLaVA-OV evaluation suite, and several aggregate numbers in the MLLM section are internally inconsistent. These issues must be resolved before the central claims can be credited.

major comments (4)
  1. [Appendix C-E] The sentence "we remove few some downstream datasets for better efficiency and robustness" does not identify which datasets were removed, how many, or the decision rule. Since the central claim is a significant improvement on the 20 reported datasets, the evaluation suite must be a fixed, pre-specified set; otherwise the reported averages in Table IV and Figure 8 could reflect post-hoc selection. Please report the full set of datasets evaluated, the removed datasets, and the results on all of them, or justify that the removals were independent of performance.
  2. [Tables XI-XIII] The aggregate numbers in the MLLM section are internally inconsistent. For the Baseline row in Table XII, the six listed General values (VQAv2 73.90, OKVQA 37.18, GQA 57.66, MMVet 21.40, SEED-Bench 61.50, RealWorldQA 50.98) average to 50.44, yet Table XII reports 48.71 and Table XI reports 48.72. The Table XI overall scores (50.54/51.50) also conflict with Table IV (50.61/51.67) for the same claimed 20-dataset averages. Please recompute and reconcile all aggregate tables, or explicitly state the averaging procedure and any excluded datasets.
  3. [Section V-C, Table IV] The text states "55.21−51.67 = 1.44" and uses this to claim the synergy gain over the manager-only baseline. The arithmetic is incorrect: 55.21−51.67 = 3.54, and the manager-over-grid gain is 55.21−53.87 = 1.34, not 1.44. This error directly affects the claim that the manager and multi-grid algorithm provide an additional 1.44% synergy. Please correct the computation and the corresponding claim.
  4. [Section V-C and Tables IV, XII] The term "significantly" is used throughout, but no confidence intervals, standard deviations, or significance tests are reported. This is especially important because some per-dataset deltas are tiny (e.g., InfoVQA +0.01, OKVQA +0.10 in Table XII) and the headline gains of +1.06/+1.44 are averages over 20 datasets. Please provide multiple runs or at least a paired significance analysis for the main comparisons, and temper the word "significantly" accordingly.
minor comments (5)
  1. [Appendix A-A] The phrase "The casual nature of the representation" should be "causal nature", and "casual attention" should be "causal attention" in the same paragraph.
  2. [Appendix C-E] The phrase "remove few some downstream datasets" is ungrammatical; it should read "remove a few downstream datasets" or "remove some downstream datasets".
  3. [Figures 23-25 and Figure 7] The x-axis label is inconsistent across figures: "Unimodal Expert Index", "Uni-Modal Layer Index", and "Uni-Modal Expert Index" are used interchangeably. Please unify the terminology.
  4. [Section V-D2, Figure 10] The caption states the ablation is on 9 datasets, but the figure reports only an overall average score. Please list the 9 datasets or cite the table containing them, for reproducibility.
  5. [Appendix C-F, Table X] The table lists "2407 and 2409" for MM-LiveBench, but the text and Table XIII refer to "July" and "Sep". Please align these labels and verify the dates.

Circularity Check

0 steps flagged · score 1.0 of 10

No significant circularity; the architecture is evaluated on held-out benchmarks, with minor self-citation only.

full rationale

The paper's central claims are empirical: ManagerTower and LLaVA-OV-Manager are compared against strong baselines on fixed downstream benchmarks. The manager equations (Eq. 4-14) define a new aggregation mechanism, and no fitted parameter is renamed as a prediction. Hyperparameters such as N=6 and injection times are tuned through ablations (Fig. 6, Fig. 10), while the headline results are reported on held-out evaluation sets. The paper cites its own conference version [1] and BridgeTower [7], a prior work by the same first author, but these are published baselines and extensions, not invoked as uniqueness theorems or as substitutes for the experiments. The undisclosed removal of 'few some downstream datasets' in Appendix C-E is a legitimate transparency and selection-bias concern, but it does not make any reported number equivalent by construction to an input; that issue belongs to correctness risk, not circularity. Overall, the derivation chain is self-contained: the manager's design is motivated by external layer-aggregation work, and its effectiveness is measured directly on benchmarks.

Assumptions & free parameters 3 free parameters · 3 assumptions · 1 invented entities

The paper relies on standard ML assumptions about pretrained encoders and adds domain-specific design choices (zero-init, number of layers, injection schedule) that were tuned on validation data. The most consequential assumption is the representativeness of the 20-dataset evaluation set, which is weakened by the vague dataset removal description.

free parameters (3)
  • Number of top unimodal layers used (N) = 6 (default, tuned on VQAv2/Flickr30K)
    The paper shows performance peaks around N=6 and degrades for N>6, indicating this value was selected based on validation performance.
  • Manager injection times in MLLM = 6 with interval 4
    Ablation in Fig. 10 shows that 6 injections balances performance and compute; injecting more gives only a small gain.
  • Zero-init with multiplicative jitter in MLLM SAUM = zero init, epsilon ~ U(0.98,1.02)
    The paper states the original SAUM did not work well on the pretrained LLM, so they introduced this design after preliminary experiments.
assumptions (3)
  • domain assumption Pretrained unimodal encoders (CLIP-ViT, RoBERTa, SigLIP) encode hierarchical semantic knowledge across layers.
    The entire manager design assumes that different layers of the encoder contain useful, distinct levels of semantic information, as argued in Section VI-B.
  • domain assumption Injecting aggregated multi-layer visual features into a pretrained LLM does not catastrophically disrupt its learned representations.
    The SAUM variant for MLLMs uses zero initialization and jitter specifically to minimize interference, reflecting the fragility of this assumption, as discussed in Section V-B2.
  • ad hoc to paper The selected 20 downstream datasets, after unspecified removals, are a representative sample of multimodal capabilities.
    Appendix C-E states 'we remove few some downstream datasets for better efficiency and robustness' without specifying which ones, which is an ad hoc selection that could bias the reported averages.
invented entities (1)
  • Manager module (SAM/SAUM/AAUM) independent evidence
    purpose: Adaptively aggregate multi-layer unimodal representations and inject them into cross-modal layers or LLM layers
    The manager's effectiveness is evaluated empirically on 24 downstream datasets, providing external falsifiable evidence. It is an architectural module, not a new physical entity.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Manager: Aggregating Insights from Unimodal Experts in Two-Tower VLMs and MLLMs." pith.science (2026). https://pith.science/paper/2I2GSVDK

@misc{pith2026250611515,
  author       = {Pith},
  title        = {Pith review of: Manager: Aggregating Insights from Unimodal Experts in Two-Tower VLMs and MLLMs},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/2I2GSVDK}},
  note         = {Machine review of arXiv:2506.11515}
}
read the original abstract

Two-Tower Vision--Language Models (VLMs) have demonstrated strong performance across various downstream VL tasks. While BridgeTower further enhances performance by building bridges between encoders, it \textit{(i)} suffers from ineffective layer-by-layer utilization of unimodal representations, \textit{(ii)} restricts the flexible exploitation of different levels of unimodal semantic knowledge, and \textit{(iii)} is limited to the evaluation on traditional low-resolution datasets only with the Two-Tower VLM architecture. In this work, we propose Manager, a lightweight, efficient and effective plugin that adaptively aggregates insights from different levels of pre-trained unimodal experts to facilitate more comprehensive VL alignment and fusion. First, under the Two-Tower VLM architecture, we introduce ManagerTower, a novel VLM that introduces the manager in each cross-modal layer. Whether with or without VL pre-training, ManagerTower outperforms previous strong baselines and achieves superior performance on 4 downstream VL tasks. Moreover, we extend our exploration to the latest Multimodal Large Language Model (MLLM) architecture. We demonstrate that LLaVA-OV-Manager significantly boosts the zero-shot performance of LLaVA-OV across different categories of capabilities, images, and resolutions on 20 downstream datasets, whether the multi-grid algorithm is enabled or not. In-depth analysis reveals that both our manager and the multi-grid algorithm can be viewed as a plugin that improves the visual representation by capturing more diverse visual details from two orthogonal perspectives (depth and width). Their synergy can mitigate the semantic ambiguity caused by the multi-grid algorithm and further improve performance. Code and models are available at https://github.com/LooperXX/ManagerTower.

Figures

Figures reproduced from arXiv: 2506.11515 by the authors.

Figure 1
Figure 1. A brief overview of BridgeTower and ManagerTower. Hollow [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Brief illustrations of LLaVA-OV-Manager. The base image and grids [PITH_FULL_IMAGE:figures/full_fig_p002_2.png] view at source ↗
Figure 3
Figure 3. An illustration of ManagerTower shows that each cross-modal layer includes a textual manager and a visual manager. Top [PITH_FULL_IMAGE:figures/full_fig_p003_3.png] view at source ↗
Figures from the paper (21 more)
Figure 4
Figure 4. Figure 4: Cosine similarity between the aggregated unimodal/cross-modal [PITH_FULL_IMAGE:figures/full_fig_p003_4.png]
Figure 5
Figure 5. Figure 5: An illustration of how the aggregated unimodal representations [PITH_FULL_IMAGE:figures/full_fig_p004_5.png]
Figure 6
Figure 6. Figure 6: VQAv2 Test-Dev Performance using different numbers of unimodal [PITH_FULL_IMAGE:figures/full_fig_p005_6.png]
Figure 7
Figure 7. Figure 7: A visualization of aggregation weights of textual and visual AAUMs in each cross-modal layer after VLP. The X-axis shows the index of the unimodal [PITH_FULL_IMAGE:figures/full_fig_p006_7.png]
Figure 8
Figure 8. Figure 8: Zero-shot performance of four baselines on [PITH_FULL_IMAGE:figures/full_fig_p008_8.png]
Figure 12
Figure 12. Figure 12: Zero-shot performance of four baselines on DocVQA validation set. [PITH_FULL_IMAGE:figures/full_fig_p009_12.png]
Figure 11
Figure 11. Figure 11: Ablation study of how manager works with multi-grid on [PITH_FULL_IMAGE:figures/full_fig_p009_11.png]
Figure 14
Figure 14. Figure 14: Zero-shot performance of four baselines on SEED-Bench. [PITH_FULL_IMAGE:figures/full_fig_p010_14.png]
Figure 17
Figure 17. Figure 17: Case studies of four baselines on OCRBench. [PITH_FULL_IMAGE:figures/full_fig_p010_17.png]
Figure 18
Figure 18. Figure 18: Cosine similarity between output representations of consecutive [PITH_FULL_IMAGE:figures/full_fig_p011_18.png]
Figure 19
Figure 19. Figure 19: Average entropy of attention weight distributions in each layer. [PITH_FULL_IMAGE:figures/full_fig_p011_19.png]
Figure 21
Figure 21. Figure 21: (a) – (d) are four main categories of the Two-Tower VLM; (e)&(f) gives a brief illustration of BridgeTower and ManagerTower. VE, TE, and CE [PITH_FULL_IMAGE:figures/full_fig_p016_21.png]
Figure 22
Figure 22. Figure 22: Brief illustrations of BridgeTower and our ManagerTower with [PITH_FULL_IMAGE:figures/full_fig_p019_22.png]
Figure 23
Figure 23. Figure 23: A visualization of aggregation weights of textual and visual SAMs in each cross-modal layer. The X-axis is the index of the unimodal expert, and [PITH_FULL_IMAGE:figures/full_fig_p020_23.png]
Figure 24
Figure 24. Figure 24: A visualization of aggregation weights of textual and visual SAUMs in each cross-modal layer. The X-axis is the index of the unimodal expert, [PITH_FULL_IMAGE:figures/full_fig_p020_24.png]
Figure 25
Figure 25. Figure 25: A visualization of aggregation weights of textual and visual AAUMs in each cross-modal layer. The X-axis is the index of the unimodal expert, [PITH_FULL_IMAGE:figures/full_fig_p020_25.png]
Figure 26
Figure 26. Figure 26: A brief illustrations of the multi-grid algorithm used in LLaVA-OV. The bilinear interpolation operation is omitted for simplicity, which is used [PITH_FULL_IMAGE:figures/full_fig_p022_26.png]
Figure 29
Figure 29. Figure 29: Case studies of four baselines on ScienceQA-IMG test set. Red and [PITH_FULL_IMAGE:figures/full_fig_p023_29.png]
Figure 28
Figure 28. Figure 28: Zero-shot performance of four baselines on ScienceQA-IMG test set. [PITH_FULL_IMAGE:figures/full_fig_p023_28.png]
Figure 33
Figure 33. Figure 33: Average KL divergence between attention weight distributions of [PITH_FULL_IMAGE:figures/full_fig_p024_33.png]
Figure 32
Figure 32. Figure 32: Average entropy of attention weight distributions in each layer of [PITH_FULL_IMAGE:figures/full_fig_p024_32.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

143 extracted references · 45 canonical work pages

  1. [1]

    ManagerTower: Aggregating the insights of uni-modal experts for vision-language representation learning,

    X. Xu, B. Li, C. Wu, S.-Y . Tseng, A. Bhiwandiwalla, S. Rosenman, V . Lal, W. Che, and N. Duan, “ManagerTower: Aggregating the insights of uni-modal experts for vision-language representation learning,” in Proc. of ACL, 2023, pp. 14 507–14 525

    2023

  2. [2]

    Making the V in VQA matter: Elevating the role of image understanding in visual question answering,

    Y . Goyal, T. Khot, D. Summers-Stay, D. Batra, and D. Parikh, “Making the V in VQA matter: Elevating the role of image understanding in visual question answering,” inProc. of CVPR, 2017, pp. 6325–6334

    2017

  3. [3]

    Visual entailment: A novel task for fine-grained image understanding,

    N. Xie, F. Lai, D. Doran, and A. Kadav, “Visual entailment: A novel task for fine-grained image understanding,”ArXiv preprint, vol. abs/1901.06706, 2019

    arXiv 1901

  4. [4]

    A corpus for reasoning about natural language grounded in photographs,

    A. Suhr, S. Zhou, A. Zhang, I. Zhang, H. Bai, and Y . Artzi, “A corpus for reasoning about natural language grounded in photographs,” in Proc. of ACL, 2019, pp. 6418–6428

    2019

  5. [5]

    From image descriptions to visual denotations: New similarity metrics for semantic inference over event descriptions,

    P. Young, A. Lai, M. Hodosh, and J. Hockenmaier, “From image descriptions to visual denotations: New similarity metrics for semantic inference over event descriptions,”Transactions of the Association for Computational Linguistics, vol. 2, pp. 67–78, 2014

    2014

  6. [6]

    An empirical study of training end-to-end vision-and-language transformers,

    Z. Dou, Y . Xu, Z. Gan, J. Wang, S. Wang, L. Wang, C. Zhu, P. Zhang, L. Yuan, N. Peng, Z. Liu, and M. Zeng, “An empirical study of training end-to-end vision-and-language transformers,” inProc. of CVPR, 2022, pp. 18 145–18 155

    2022

  7. [7]

    Bridgetower: Building bridges between encoders in vision-language representation learning,

    X. Xu, C. Wu, S. Rosenman, V . Lal, W. Che, and N. Duan, “Bridgetower: Building bridges between encoders in vision-language representation learning,” inProc. of AAAI, 2023, pp. 10 637–10 647

    2023

  8. [8]

    Learning transferable visual models from natural language supervi- sion,

    A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark, G. Krueger, and I. Sutskever, “Learning transferable visual models from natural language supervi- sion,” inProc. of ICML, vol. 139, 2021, pp. 8748–8763

    2021

Show all 143 references
  1. [9]

    Roberta: A robustly optimized bert pretraining approach,

    Y . Liu, M. Ott, N. Goyal, J. Du, M. Joshi, D. Chen, O. Levy, M. Lewis, L. Zettlemoyer, and V . Stoyanov, “Roberta: A robustly optimized bert pretraining approach,”ArXiv preprint, vol. abs/1907.11692, 2019

    1907 arXiv

  2. [10]

    Learning deep transformer models for machine translation,

    Q. Wang, B. Li, T. Xiao, J. Zhu, C. Li, D. F. Wong, and L. S. Chao, “Learning deep transformer models for machine translation,” inProc. of ACL, 2019, pp. 1810–1822

    2019

  3. [11]

    Llava-onevision: Easy visual task transfer,

    B. Li, Y . Zhang, D. Guo, R. Zhang, F. Li, H. Zhang, K. Zhang, Y . Li, Z. Liu, and C. Li, “Llava-onevision: Easy visual task transfer,”ArXiv preprint, vol. abs/2408.03326, 2024

    2024 arXiv

  4. [12]

    Llava- next: Improved reasoning, ocr, and world knowledge,

    H. Liu, C. Li, Y . Li, B. Li, Y . Zhang, S. Shen, and Y . J. Lee, “Llava- next: Improved reasoning, ocr, and world knowledge,” 2024

    2024

  5. [13]

    How much can CLIP benefit vision-and-language tasks?

    S. Shen, L. H. Li, H. Tan, M. Bansal, A. Rohrbach, K. Chang, Z. Yao, and K. Keutzer, “How much can CLIP benefit vision-and-language tasks?” inProc. of ICLR, 2022

    2022

  6. [14]

    UNIMO-2: End-to-end unified vision-language grounded learning,

    W. Li, C. Gao, G. Niu, X. Xiao, H. Liu, J. Liu, H. Wu, and H. Wang, “UNIMO-2: End-to-end unified vision-language grounded learning,” in Proc. of ACL Findings, 2022, pp. 3187–3201

    2022

  7. [15]

    Neural machine translation of rare words with subword units,

    R. Sennrich, B. Haddow, and A. Birch, “Neural machine translation of rare words with subword units,” inProc. of ACL, 2016, pp. 1715–1725

    2016

  8. [16]

    Language models are unsupervised multitask learners,

    A. Radford, J. Wu, R. Child, D. Luan, D. Amodei, I. Sutskeveret al., “Language models are unsupervised multitask learners,”OpenAI blog, 2019

    2019

  9. [17]

    Attention is all you need,

    A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin, “Attention is all you need,” in Proc. of NeurIPS, 2017, pp. 5998–6008

    2017

  10. [18]

    Vilbert: Pretraining task- agnostic visiolinguistic representations for vision-and-language tasks,

    J. Lu, D. Batra, D. Parikh, and S. Lee, “Vilbert: Pretraining task- agnostic visiolinguistic representations for vision-and-language tasks,” inProc. of NeurIPS, 2019, pp. 13–23

    2019

  11. [19]

    Multi-layer representation fusion for neural machine translation,

    Q. Wang, F. Li, T. Xiao, Y . Li, Y . Li, and J. Zhu, “Multi-layer representation fusion for neural machine translation,” inProc. of COLING, 2018, pp. 3015–3026

    2018

  12. [20]

    Multiscale collaborative deep models for neural machine translation,

    X. Wei, H. Yu, Y . Hu, Y . Zhang, R. Weng, and W. Luo, “Multiscale collaborative deep models for neural machine translation,” inProc. of ACL, 2020, pp. 414–426

    2020

  13. [21]

    Layer normalization,

    J. L. Ba, J. R. Kiros, and G. E. Hinton, “Layer normalization,”ArXiv preprint, vol. abs/1607.06450, 2016

    2016 arXiv

  14. [22]

    Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity,

    W. Fedus, B. Zoph, and N. Shazeer, “Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity,”Journal of Machine Learning Research, vol. 23, pp. 1–39, 2022

    2022

  15. [23]

    Decoupled weight decay regularization,

    I. Loshchilov and F. Hutter, “Decoupled weight decay regularization,” inProc. of ICLR, 2019

    2019

  16. [24]

    Vilt: Vision-and-language transformer without convolution or region supervision,

    W. Kim, B. Son, and I. Kim, “Vilt: Vision-and-language transformer without convolution or region supervision,” inProc. of ICML, vol. 139, 2021, pp. 5583–5594

    2021

  17. [25]

    Uniter: Universal image-text representation learning,

    Y .-C. Chen, L. Li, L. Yu, A. El Kholy, F. Ahmed, Z. Gan, Y . Cheng, and J. Liu, “Uniter: Universal image-text representation learning,” in Proc. of ECCV, 2020

    2020

  18. [26]

    UNIMO: Towards unified-modal understanding and generation via cross-modal contrastive learning,

    W. Li, C. Gao, G. Niu, X. Xiao, H. Liu, J. Liu, H. Wu, and H. Wang, “UNIMO: Towards unified-modal understanding and generation via cross-modal contrastive learning,” inProc. of ACL, 2021, pp. 2592– 2607

    2021

  19. [27]

    Align before fuse: Vision and language representation learning with momentum distillation,

    J. Li, R. R. Selvaraju, A. Gotmare, S. R. Joty, C. Xiong, and S. C. Hoi, “Align before fuse: Vision and language representation learning with momentum distillation,” inProc. of NeurIPS, 2021, pp. 9694–9705

    2021

  20. [28]

    Vlmo: Unified vision-language pre- training with mixture-of-modality-experts,

    H. Bao, W. Wang, L. Dong, Q. Liu, O. K. Mohammed, K. Aggarwal, S. Som, S. Piao, and F. Wei, “Vlmo: Unified vision-language pre- training with mixture-of-modality-experts,” inProc. of NeurIPS, 2022

    2022

  21. [29]

    Simvlm: Simple visual language model pretraining with weak supervision,

    Z. Wang, J. Yu, A. W. Yu, Z. Dai, Y . Tsvetkov, and Y . Cao, “Simvlm: Simple visual language model pretraining with weak supervision,” in Proc. of ICLR, 2022

    2022

  22. [30]

    BLIP: bootstrapping language-image pre-training for unified vision-language understanding and generation,

    J. Li, D. Li, C. Xiong, and S. C. H. Hoi, “BLIP: bootstrapping language-image pre-training for unified vision-language understanding and generation,” inProc. of ICML, vol. 162, 2022, pp. 12 888–12 900

    2022

  23. [31]

    Conceptual captions: A cleaned, hypernymed, image alt-text dataset for automatic image captioning,

    P. Sharma, N. Ding, S. Goodman, and R. Soricut, “Conceptual captions: A cleaned, hypernymed, image alt-text dataset for automatic image captioning,” inProc. of ACL, 2018, pp. 2556–2565

    2018

  24. [32]

    Im2text: Describing images using 1 million captioned photographs,

    V . Ordonez, G. Kulkarni, and T. L. Berg, “Im2text: Describing images using 1 million captioned photographs,” inProc. of NeurIPS, 2011, pp. 1143–1151

    2011

  25. [33]

    Microsoft coco captions: Data collection and evaluation server,

    X. Chen, H. Fang, T.-Y . Lin, R. Vedantam, S. Gupta, P. Doll ´ar, and C. L. Zitnick, “Microsoft coco captions: Data collection and evaluation server,”ArXiv preprint, vol. abs/1504.00325, 2015

    2015 arXiv

  26. [34]

    Visual genome: Connect- ing language and vision using crowdsourced dense image annotations,

    R. Krishna, Y . Zhu, O. Groth, J. Johnson, K. Hata, J. Kravitz, S. Chen, Y . Kalantidis, L.-J. Li, D. A. Shammaet al., “Visual genome: Connect- ing language and vision using crowdsourced dense image annotations,” International journal of computer vision, 2017

    2017

  27. [35]

    Improved baselines with visual instruction tuning,

    H. Liu, C. Li, Y . Li, and Y . J. Lee, “Improved baselines with visual instruction tuning,” inProc. of CVPR, 2024, pp. 26 296–26 306

    2024

  28. [36]

    Monkey: Image resolution and text label are important things for large multi-modal models,

    Z. Li, B. Yang, Q. Liu, Z. Ma, S. Zhang, J. Yang, Y . Sun, Y . Liu, and X. Bai, “Monkey: Image resolution and text label are important things for large multi-modal models,” inProc. of CVPR, 2024, pp. 26 763–26 773

    2024

  29. [37]

    Docvqa: A dataset for vqa on document images,

    M. Mathew, D. Karatzas, and C. Jawahar, “Docvqa: A dataset for vqa on document images,” inProceedings of the IEEE/CVF winter conference on applications of computer vision, 2021, pp. 2200–2209

    2021

  30. [38]

    Ocrbench: On the hidden mystery of ocr in large multimodal models,

    Y . Liu, Z. Li, M. Huang, B. Yang, W. Yu, C. Li, X. Yin, C. lin Liu, L. Jin, and X. Bai, “Ocrbench: On the hidden mystery of ocr in large multimodal models,”ArXiv preprint, vol. abs/2305.07895, 2023

    2023 arXiv

  31. [39]

    Sigmoid loss for language image pre-training,

    X. Zhai, B. Mustafa, A. Kolesnikov, and L. Beyer, “Sigmoid loss for language image pre-training,” inProc. of ICCV, 2023, pp. 11 941– 11 952

    2023

  32. [40]

    Qwen2 technical report,

    A. Yang, B. Yang, B. Hui, B. Zheng, B. Yu, C. Zhou, C. Li, C. Li, D. Liu, F. Huanget al., “Qwen2 technical report,”ArXiv preprint, vol. abs/2407.10671, 2024

    2024 arXiv

  33. [41]

    Exploring plain vision transformer backbones for object detection,

    Y . Li, H. Mao, R. Girshick, and K. He, “Exploring plain vision transformer backbones for object detection,”ArXiv preprint, vol. abs/2203.16527, 2022

    2022 arXiv

  34. [42]

    Llama-adapter: Efficient fine-tuning of language models with zero-init attention,

    R. Zhang, J. Han, C. Liu, P. Gao, A. Zhou, X. Hu, S. Yan, P. Lu, H. Li, and Y . Qiao, “Llama-adapter: Efficient fine-tuning of language models with zero-init attention,”ArXiv preprint, vol. abs/2303.16199, 2023

    2023 arXiv

  35. [43]

    OK-VQA: A visual question answering benchmark requiring external knowledge,

    K. Marino, M. Rastegari, A. Farhadi, and R. Mottaghi, “OK-VQA: A visual question answering benchmark requiring external knowledge,” inProc. of CVPR, 2019, pp. 3195–3204

    2019

  36. [44]

    GQA: A new dataset for real-world visual reasoning and compositional question answering,

    D. A. Hudson and C. D. Manning, “GQA: A new dataset for real-world visual reasoning and compositional question answering,” inProc. of CVPR, 2019, pp. 6700–6709

    2019

  37. [45]

    MM-vet: Evaluating large multimodal models for integrated capabilities,

    W. Yu, Z. Yang, L. Li, J. Wang, K. Lin, Z. Liu, X. Wang, and L. Wang, “MM-vet: Evaluating large multimodal models for integrated capabilities,” inProc. of ICML, 2024

    2024

  38. [46]

    Seed-bench: Benchmarking multimodal llms with generative comprehension,

    B. Li, R. Wang, G. Wang, Y . Ge, Y . Ge, and Y . Shan, “Seed-bench: Benchmarking multimodal llms with generative comprehension,”ArXiv preprint, vol. abs/2307.16125, 2023

    2023 arXiv

  39. [47]

    Grok-1.5 vision preview

    x.ai, “Grok-1.5 vision preview.” 2019

    2019

  40. [48]

    Towards VQA models that can read,

    A. Singh, V . Natarajan, M. Shah, Y . Jiang, X. Chen, D. Batra, D. Parikh, and M. Rohrbach, “Towards VQA models that can read,” inProc. of CVPR, 2019, pp. 8317–8326

    2019

  41. [49]

    ChartQA: A benchmark for question answering about charts with visual and logical reasoning,

    A. Masry, X. L. Do, J. Q. Tan, S. Joty, and E. Hoque, “ChartQA: A benchmark for question answering about charts with visual and logical reasoning,” inProc. of ACL Findings, 2022, pp. 2263–2279

    2022

  42. [50]

    Infographicvqa,

    M. Mathew, V . Bagal, R. Tito, D. Karatzas, E. Valveny, and C. Jawahar, “Infographicvqa,” inProceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, 2022, pp. 1697–1706. IEEE TRANSACTIONS ON CIRCUITS AND SYSTEMS FOR VIDEO TECHNOLOGY 29

    2022

  43. [51]

    A diagram is worth a dozen images,

    A. Kembhavi, M. Salvato, E. Kolve, M. Seo, H. Hajishirzi, and A. Farhadi, “A diagram is worth a dozen images,” inProc. of ECCV, 2016, pp. 235–251

    2016

  44. [52]

    Learn to explain: Multimodal reasoning via thought chains for science question answering,

    P. Lu, S. Mishra, T. Xia, L. Qiu, K. Chang, S. Zhu, O. Tafjord, P. Clark, and A. Kalyan, “Learn to explain: Multimodal reasoning via thought chains for science question answering,” inProc. of NeurIPS, 2022

    2022

  45. [53]

    Mmmu: A massive multi-discipline multimodal understanding and reasoning benchmark for expert agi,

    X. Yue, Y . Ni, K. Zhang, T. Zheng, R. Liu, G. Zhang, S. Stevens, D. Jiang, W. Ren, Y . Sun, C. Wei, B. Yu, R. Yuan, R. Sun, M. Yin, B. Zheng, Z. Yang, Y . Liu, W. Huang, H. Sun, Y . Su, and W. Chen, “Mmmu: A massive multi-discipline multimodal understanding and reasoning benc...

    2024

  46. [54]

    Mathvista: Evaluating mathematical reasoning of foundation models in visual contexts,

    P. Lu, H. Bansal, T. Xia, J. Liu, C. Li, H. Hajishirzi, H. Cheng, K.-W. Chang, M. Galley, and J. Gao, “Mathvista: Evaluating mathematical reasoning of foundation models in visual contexts,” inProc. of ICLR, 2024

    2024

  47. [55]

    Llava-next: What else influences visual instruction tuning beyond data?

    B. Li, H. Zhang, K. Zhang, D. Guo, Y . Zhang, R. Zhang, F. Li, Z. Liu, and C. Li, “Llava-next: What else influences visual instruction tuning beyond data?” 2024

    2024

  48. [56]

    Lmms-eval: Reality check on the evaluation of large multimodal models,

    K. Zhang, B. Li, P. Zhang, F. Pu, J. A. Cahyono, K. Hu, S. Liu, Y . Zhang, J. Yang, C. Liet al., “Lmms-eval: Reality check on the evaluation of large multimodal models,”ArXiv preprint, vol. abs/2407.12772, 2024

    2024 arXiv

  49. [57]

    Visual instruction tuning,

    H. Liu, C. Li, Q. Wu, and Y . J. Lee, “Visual instruction tuning,” in Proc. of NeurIPS, 2023

    2023

  50. [58]

    An image is worth 16x16 words: Transformers for image recognition at scale,

    A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly, J. Uszkoreit, and N. Houlsby, “An image is worth 16x16 words: Transformers for image recognition at scale,” inProc. of ICLR, 2021

    2021

  51. [59]

    Mini-monkey: Alleviating the semantic sawtooth effect for lightweight mllms via complementary image pyramid,

    M. Huang, Y . Liu, D. Liang, L. Jin, and X. Bai, “Mini-monkey: Alleviating the semantic sawtooth effect for lightweight mllms via complementary image pyramid,”ArXiv preprint, vol. abs/2408.02034, 2024

    2024 arXiv

  52. [60]

    Neural machine translation by jointly learning to align and translate,

    D. Bahdanau, K. Cho, and Y . Bengio, “Neural machine translation by jointly learning to align and translate,” inProc. of ICLR, 2015

    2015

  53. [61]

    Revealing the dark secrets of masked image modeling,

    Z. Xie, Z. Geng, J. Hu, Z. Zhang, H. Hu, and Y . Cao, “Revealing the dark secrets of masked image modeling,” inProc. of CVPR, 2023, pp. 14 475–14 485

    2023

  54. [62]

    On information and sufficiency,

    S. Kullback and R. A. Leibler, “On information and sufficiency,”The annals of mathematical statistics, vol. 22, pp. 79–86, 1951

    1951

  55. [63]

    VL-BERT: pre-training of generic visual-linguistic representations,

    W. Su, X. Zhu, Y . Cao, B. Li, L. Lu, F. Wei, and J. Dai, “VL-BERT: pre-training of generic visual-linguistic representations,” inProc. of ICLR, 2020

    2020

  56. [64]

    Unicoder-vl: A universal encoder for vision and language by cross-modal pre-training,

    G. Li, N. Duan, Y . Fang, M. Gong, and D. Jiang, “Unicoder-vl: A universal encoder for vision and language by cross-modal pre-training,” inProc. of AAAI, 2020, pp. 11 336–11 344

    2020

  57. [65]

    Oscar: Object-semantics aligned pre-training for vision-language tasks,

    X. Li, X. Yin, C. Li, P. Zhang, X. Hu, L. Zhang, L. Wang, H. Hu, L. Dong, F. Weiet al., “Oscar: Object-semantics aligned pre-training for vision-language tasks,” inProc. of ECCV, 2020

    2020

  58. [66]

    Unifying architectures, tasks, and modalities through a simple sequence-to-sequence learning framework,

    P. Wang, A. Yang, R. Men, J. Lin, S. Bai, Z. Li, J. Ma, C. Zhou, J. Zhou, and H. Yang, “Unifying architectures, tasks, and modalities through a simple sequence-to-sequence learning framework,”ArXiv preprint, vol. abs/2202.03052, 2022

    2022 arXiv

  59. [67]

    Image as a foreign language: Beit pretraining for all vision and vision-language tasks,

    W. Wang, H. Bao, L. Dong, J. Bjorck, Z. Peng, Q. Liu, K. Aggarwal, O. K. Mohammed, S. Singhal, S. Somet al., “Image as a foreign language: Beit pretraining for all vision and vision-language tasks,” ArXiv preprint, vol. abs/2208.10442, 2022

    2022 arXiv

  60. [68]

    Coca: Contrastive captioners are image-text foundation mod- els,

    J. Yu, Z. Wang, V . Vasudevan, L. Yeung, M. Seyedhosseini, and Y . Wu, “Coca: Contrastive captioners are image-text foundation mod- els,”ArXiv preprint, vol. abs/2205.01917, 2022

    2022 arXiv

  61. [69]

    Exploring vision-language foundation model for novel object captioning,

    J. Luo, Y . Li, Y . Pan, T. Yao, J. Feng, H. Chao, and T. Mei, “Exploring vision-language foundation model for novel object captioning,”IEEE Trans. Circuits Syst. Video Technol., pp. 1–1, 2024

    2024

  62. [70]

    Unsupervised domain adaption harnessing vision-language pre-training,

    W. Zhou and Z. Zhou, “Unsupervised domain adaption harnessing vision-language pre-training,”IEEE Trans. Circuits Syst. Video Tech- nol., vol. 34, pp. 8201–8214, 2024

    2024

  63. [71]

    Do vision transformers see like convolutional neural networks?

    M. Raghu, T. Unterthiner, S. Kornblith, C. Zhang, and A. Dosovitskiy, “Do vision transformers see like convolutional neural networks?” in Proc. of NeurIPS, 2021, pp. 12 116–12 128

    2021

  64. [72]

    Intriguing properties of vision transformers,

    M. Naseer, K. Ranasinghe, S. Khan, M. Hayat, F. S. Khan, and M. Yang, “Intriguing properties of vision transformers,” inProc. of NeurIPS, 2021, pp. 23 296–23 308

    2021

  65. [73]

    Dissecting contextual word embeddings: Architecture and representation,

    M. E. Peters, M. Neumann, L. Zettlemoyer, and W.-t. Yih, “Dissecting contextual word embeddings: Architecture and representation,” inProc. of EMNLP, 2018, pp. 1499–1509

    2018

  66. [74]

    Linguistic knowledge and transferability of contextual representa- tions,

    N. F. Liu, M. Gardner, Y . Belinkov, M. E. Peters, and N. A. Smith, “Linguistic knowledge and transferability of contextual representa- tions,” inProc. of NAACL, 2019, pp. 1073–1094

    2019

  67. [75]

    What does BERT learn about the structure of language?

    G. Jawahar, B. Sagot, and D. Seddah, “What does BERT learn about the structure of language?” inProc. of ACL, 2019, pp. 3651–3657

    2019

  68. [76]

    BERT: Pre- training of deep bidirectional transformers for language understanding,

    J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, “BERT: Pre- training of deep bidirectional transformers for language understanding,” inProc. of NAACL, 2019, pp. 4171–4186

    2019

  69. [77]

    Feature pyramid networks for object detection,

    T. Lin, P. Doll ´ar, R. B. Girshick, K. He, B. Hariharan, and S. J. Belongie, “Feature pyramid networks for object detection,” inProc. of CVPR, 2017, pp. 936–944

    2017

  70. [78]

    Densely connected convolutional networks,

    G. Huang, Z. Liu, L. van der Maaten, and K. Q. Weinberger, “Densely connected convolutional networks,” inProc. of CVPR, 2017, pp. 2261– 2269

    2017

  71. [79]

    Deep layer aggregation,

    F. Yu, D. Wang, E. Shelhamer, and T. Darrell, “Deep layer aggregation,” inProc. of CVPR, 2018, pp. 2403–2412

    2018

  72. [80]

    Segformer: Simple and efficient design for semantic segmentation with transformers,

    E. Xie, W. Wang, Z. Yu, A. Anandkumar, J. M. Alvarez, and P. Luo, “Segformer: Simple and efficient design for semantic segmentation with transformers,” inProc. of NeurIPS, 2021, pp. 12 077–12 090

    2021

  73. [81]

    Clsr: Cross-layer interaction pyramid super-resolution network,

    D. Huang, X. Zhu, X. Li, and H. Zeng, “Clsr: Cross-layer interaction pyramid super-resolution network,”IEEE Trans. Circuits Syst. Video Technol., vol. 33, pp. 6273–6287, 2023

    2023

  74. [82]

    Attention-based layer fusion and token masking for weakly supervised semantic segmentation,

    Y . Zhang and X. Zhu, “Attention-based layer fusion and token masking for weakly supervised semantic segmentation,”IEEE Trans. Circuits Syst. Video Technol., vol. 34, pp. 7912–7921, 2024

    2024

  75. [83]

    Artificial- spiking hierarchical networks for vision-language representation learn- ing,

    Y . Chen, S. Zhang, Y . Sun, J. Yang, W. Liang, and H. Wang, “Artificial- spiking hierarchical networks for vision-language representation learn- ing,”IEEE Trans. Circuits Syst. Video Technol., pp. 1–1, 2024

    2024

  76. [84]

    Deep contextualized word representations,

    M. E. Peters, M. Neumann, M. Iyyer, M. Gardner, C. Clark, K. Lee, and L. Zettlemoyer, “Deep contextualized word representations,” inProc. of NAACL, 2018, pp. 2227–2237

    2018

  77. [85]

    Coarse-to-fine vision-language pre-training with fusion in the backbone,

    Z. Dou, A. Kamath, Z. Gan, P. Zhang, J. Wang, L. Li, Z. Liu, C. Liu, Y . LeCun, N. Peng, J. Gao, and L. Wang, “Coarse-to-fine vision-language pre-training with fusion in the backbone,” inProc. of NeurIPS, 2022

    2022

  78. [86]

    Dense connector for mllms,

    H. Yao, W. Wu, T. Yang, Y . Song, M. Zhang, H. Feng, Y . Sun, Z. Li, W. Ouyang, and J. Wang, “Dense connector for mllms,”ArXiv preprint, vol. abs/2405.13800, 2024

    2024 arXiv

  79. [87]

    Tokenpacker: Efficient visual projector for multimodal llm,

    W. Li, Y . Yuan, J. Liu, D. Tang, S. Wang, J. Qin, J. Zhu, and L. Zhang, “Tokenpacker: Efficient visual projector for multimodal llm,”ArXiv preprint, vol. abs/2407.02392, 2024

    2024 arXiv

  80. [88]

    Language models are few-shot learners,

    T. B. Brown, B. Mann, N. Ryder, M. Subbiah, J. Kaplan, P. Dhari- wal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, S. Agarwal, A. Herbert-V oss, G. Krueger, T. Henighan, R. Child, A. Ramesh, D. M. Ziegler, J. Wu, C. Winter, C. Hesse, M. Chen, E. Sigler, M. Litwin, S. Gray, ...

    2020

  81. [89]

    Llama 2: Open foundation and fine-tuned chat models,

    H. Touvron, L. Martin, K. Stone, P. Albert, A. Almahairi, Y . Babaei, N. Bashlykov, S. Batra, P. Bhargava, S. Bhosaleet al., “Llama 2: Open foundation and fine-tuned chat models,”ArXiv preprint, vol. abs/2307.09288, 2023

    2023 arXiv

  82. [90]

    Large language models meet nlp: A survey,

    L. Qin, Q. Chen, X. Feng, Y . Wu, Y . Zhang, Y . Li, M. Li, W. Che, and P. S. Yu, “Large language models meet nlp: A survey,”ArXiv preprint, vol. abs/2405.12819, 2024

    2024 arXiv

  83. [91]

    BLIP-2: bootstrapping language-image pre-training with frozen image encoders and large language models,

    J. Li, D. Li, S. Savarese, and S. C. H. Hoi, “BLIP-2: bootstrapping language-image pre-training with frozen image encoders and large language models,” inProc. of ICML, vol. 202, 2023, pp. 19 730–19 742

    2023

  84. [92]

    Introducing our multimodal models,

    R. Bavishi, E. Elsen, C. Hawthorne, M. Nye, A. Odena, A. Somani, and S. Tas ¸ırlar, “Introducing our multimodal models,” 2023

    2023

  85. [93]

    Mini-gemini: Mining the potential of multi-modality vision language models,

    Y . Li, Y . Zhang, C. Wang, Z. Zhong, Y . Chen, R. Chu, S. Liu, and J. Jia, “Mini-gemini: Mining the potential of multi-modality vision language models,”ArXiv preprint, vol. abs/2403.18814, 2024

    2024 arXiv

  86. [94]

    Qwen2-vl: Enhancing vision-language model’s perception of the world at any resolution,

    P. Wang, S. Bai, S. Tan, S. Wang, Z. Fan, J. Bai, K. Chen, X. Liu, J. Wang, W. Geet al., “Qwen2-vl: Enhancing vision-language model’s perception of the world at any resolution,”ArXiv preprint, vol. abs/2409.12191, 2024

    2024 arXiv

  87. [95]

    Oryx mllm: On- demand spatial-temporal understanding at arbitrary resolution,

    Z. Liu, Y . Dong, Z. Liu, W. Hu, J. Lu, and Y . Rao, “Oryx mllm: On- demand spatial-temporal understanding at arbitrary resolution,”ArXiv preprint, vol. abs/2409.12961, 2024

    2024 arXiv

  88. [96]

    Sphinx: The joint mixing of weights, tasks, and visual embeddings for multi-modal large language models,

    Z. Lin, C. Liu, R. Zhang, P. Gao, L. Qiu, H. Xiao, H. Qiu, C. Lin, W. Shao, K. Chenet al., “Sphinx: The joint mixing of weights, tasks, and visual embeddings for multi-modal large language models,”ArXiv preprint, vol. abs/2311.07575, 2023

    2023 arXiv

  89. [97]

    When do we not need larger vision models?

    B. Shi, Z. Wu, M. Mao, X. Wang, and T. Darrell, “When do we not need larger vision models?” inProc. of ECCV, 2025, pp. 444–462

    2025

  90. [98]

    Textmonkey: An ocr-free large multimodal model for understanding document,

    Y . Liu, B. Yang, Q. Liu, Z. Li, Z. Ma, S. Zhang, and X. Bai, “Textmonkey: An ocr-free large multimodal model for understanding document,”ArXiv preprint, vol. abs/2403.04473, 2024. IEEE TRANSACTIONS ON CIRCUITS AND SYSTEMS FOR VIDEO TECHNOLOGY 30

    2024 arXiv

  91. [99]

    Honeybee: Locality-enhanced projector for multimodal llm,

    J. Cha, W. Kang, J. Mun, and B. Roh, “Honeybee: Locality-enhanced projector for multimodal llm,” inProc. of CVPR, 2024, pp. 13 817– 13 827

    2024

  92. [100]

    Unified language model pre-training for natural language understanding and generation,

    L. Dong, N. Yang, W. Wang, F. Wei, X. Liu, Y . Wang, J. Gao, M. Zhou, and H. Hon, “Unified language model pre-training for natural language understanding and generation,” inProc. of NeurIPS, 2019, pp. 13 042– 13 054

    2019

  93. [101]

    Language models are general-purpose interfaces,

    Y . Hao, H. Song, L. Dong, S. Huang, Z. Chi, W. Wang, S. Ma, and F. Wei, “Language models are general-purpose interfaces,”ArXiv preprint, vol. abs/2206.06336, 2022

    2022 arXiv

  94. [102]

    Prismer: A vision-language model with multi-task experts,

    S. Liu, L. Fan, E. Johns, Z. Yu, C. Xiao, and A. Anandkumar, “Prismer: A vision-language model with multi-task experts,”Transactions on Machine Learning Research, 2024

    2024

  95. [103]

    Flamingo: a visual language model for few-shot learning,

    J. Alayrac, J. Donahue, P. Luc, A. Miech, I. Barr, Y . Hasson, K. Lenc, A. Mensch, K. Millican, M. Reynolds, R. Ring, E. Rutherford, S. Cabi, T. Han, Z. Gong, S. Samangooei, M. Monteiro, J. L. Menick, S. Borgeaud, A. Brock, A. Nematzadeh, S. Sharifzadeh, M. Binkowski, R. Barre...

    2022

  96. [104]

    Cogvlm: Visual expert for pretrained language models,

    W. Wang, Q. Lv, W. Yu, W. Hong, J. Qi, Y . Wang, J. Ji, Z. Yang, L. Zhao, X. Songet al., “Cogvlm: Visual expert for pretrained language models,”ArXiv preprint, vol. abs/2311.03079, 2023

    2023 arXiv

  97. [105]

    MMICL: Empowering vision-language model with multi-modal in-context learning,

    H. Zhao, Z. Cai, S. Si, X. Ma, K. An, L. Chen, Z. Liu, S. Wang, W. Han, and B. Chang, “MMICL: Empowering vision-language model with multi-modal in-context learning,” inProc. of ICLR, 2024

    2024

  98. [106]

    What factors affect multi-modal in-context learning? an in-depth exploration,

    L. Qin, Q. Chen, H. Fei, Z. Chen, M. Li, and W. Che, “What factors affect multi-modal in-context learning? an in-depth exploration,”ArXiv preprint, vol. abs/2410.20482, 2024

    2024 arXiv

  99. [107]

    Mul- timodal chain-of-thought reasoning in language models,

    Z. Zhang, A. Zhang, M. Li, hai zhao, G. Karypis, and A. Smola, “Mul- timodal chain-of-thought reasoning in language models,”Transactions on Machine Learning Research, 2024

    2024

  100. [108]

    M 3CoT: A novel benchmark for multi-domain multi-step multi-modal chain-of- thought,

    Q. Chen, L. Qin, J. Zhang, Z. Chen, X. Xu, and W. Che, “M 3CoT: A novel benchmark for multi-domain multi-step multi-modal chain-of- thought,” inProc. of ACL, 2024, pp. 8199–8221

    2024

  101. [109]

    Multi-grained vision language pre- training: Aligning texts with visual concepts,

    Y . Zeng, X. Zhang, and H. Li, “Multi-grained vision language pre- training: Aligning texts with visual concepts,” inProc. of ICML, vol. 162, 2022, pp. 25 994–26 009

    2022

  102. [110]

    Exploring multi-grained concept annotations for multimodal large language mod- els,

    X. Xu, T. Niu, Y . Xie, L. Qin, W. Che, and M.-Y . Kan, “Exploring multi-grained concept annotations for multimodal large language mod- els,”ArXiv preprint, vol. abs/2412.05939, 2024

    2024 arXiv

  103. [111]

    Eagle: Exploring the design space for multimodal llms with mixture of encoders,

    M. Shi, F. Liu, S. Wang, S. Liao, S. Radhakrishnan, D.-A. Huang, H. Yin, K. Sapra, Y . Yacoob, H. Shiet al., “Eagle: Exploring the design space for multimodal llms with mixture of encoders,”ArXiv preprint, vol. abs/2408.15998, 2024

    2024 arXiv

  104. [112]

    Cambrian-1: A fully open, vision-centric exploration of multimodal LLMs,

    S. Tong, E. L. B. II, P. Wu, S. Woo, A. J. IYER, S. C. Akula, S. Yang, J. Yang, M. Middepogu, Z. Wang, X. Pan, R. Fergus, Y . LeCun, and S. Xie, “Cambrian-1: A fully open, vision-centric exploration of multimodal LLMs,” inProc. of NeurIPS, 2024

    2024

  105. [113]

    DINOv2: Learning robust visual features without supervision,

    M. Oquab, T. Darcet, T. Moutakanni, H. V . V o, M. Szafraniec, V . Khalidov, P. Fernandez, D. HAZIZA, F. Massa, A. El-Nouby, M. Assran, N. Ballas, W. Galuba, R. Howes, P.-Y . Huang, S.-W. Li, I. Misra, M. Rabbat, V . Sharma, G. Synnaeve, H. Xu, H. Jegou, J. Mairal, P. Labatut,...

    2024

  106. [114]

    Nvlm: Open frontier-class multimodal llms,

    W. Dai, N. Lee, B. Wang, Z. Yang, Z. Liu, J. Barker, T. Rintamaki, M. Shoeybi, B. Catanzaro, and W. Ping, “Nvlm: Open frontier-class multimodal llms,”ArXiv preprint, vol. abs/2409.11402, 2024

    2024 arXiv

  107. [115]

    Expanding performance boundaries of open- source multimodal models with model, data, and test-time scaling,

    Z. Chen, W. Wang, Y . Cao, Y . Liu, Z. Gao, E. Cui, J. Zhu, S. Ye, H. Tian, Z. Liuet al., “Expanding performance boundaries of open- source multimodal models with model, data, and test-time scaling,” ArXiv preprint, vol. abs/2412.05271, 2024

    2024 arXiv

  108. [116]

    Deepseek-vl2: Mixture-of-experts vision- language models for advanced multimodal understanding,

    Z. Wu, X. Chen, Z. Pan, X. Liu, W. Liu, D. Dai, H. Gao, Y . Ma, C. Wu, B. Wang, Z. Xie, Y . Wu, K. Hu, J. Wang, Y . Sun, Y . Li, Y . Piao, K. Guan, A. Liu, X. Xie, Y . You, K. Dong, X. Yu, H. Zhang, L. Zhao, Y . Wang, and C. Ruan, “Deepseek-vl2: Mixture-of-experts vision- lang...

    2024

  109. [117]

    Hires-llava: Restoring fragmentation input in high-resolution large vision-language models,

    R. Huang, X. Ding, C. Wang, J. Han, Y . Liu, H. Zhao, H. Xu, L. Hou, W. Zhang, and X. Liang, “Hires-llava: Restoring fragmentation input in high-resolution large vision-language models,”ArXiv preprint, vol. abs/2407.08706, 2024

    2024

  110. [118]

    Scaling up visual and vision-language representation learning with noisy text supervision,

    C. Jia, Y . Yang, Y . Xia, Y . Chen, Z. Parekh, H. Pham, Q. V . Le, Y . Sung, Z. Li, and T. Duerig, “Scaling up visual and vision-language representation learning with noisy text supervision,” inProc. of ICML, vol. 139, 2021, pp. 4904–4916

    2021

  111. [119]

    Visual- bert: Asimple and performant baseline for vision and language,

    L. H. Li, M. Yatskar, D. Yin, C.-J. Hsieh, and K.-W. Chang, “Visual- bert: Asimple and performant baseline for vision and language,”ArXiv preprint, vol. abs/1908.03557, 2019

    1908 arXiv

  112. [120]

    Unified vision-language pre-training for image captioning and VQA,

    L. Zhou, H. Palangi, L. Zhang, H. Hu, J. J. Corso, and J. Gao, “Unified vision-language pre-training for image captioning and VQA,” inProc. of AAAI, 2020, pp. 13 041–13 049

    2020

  113. [121]

    Vinvl: Revisiting visual representations in vision-language models,

    P. Zhang, X. Li, X. Hu, J. Yang, L. Zhang, L. Wang, Y . Choi, and J. Gao, “Vinvl: Revisiting visual representations in vision-language models,” inProc. of CVPR, 2021, pp. 5579–5588

    2021

  114. [122]

    Unifying vision-and-language tasks via text generation,

    J. Cho, J. Lei, H. Tan, and M. Bansal, “Unifying vision-and-language tasks via text generation,” inProc. of ICML, vol. 139, 2021, pp. 1931– 1942

    2021

  115. [123]

    Pixel-bert: Aligning image pixels with text by deep multi-modal transformers,

    Z. Huang, Z. Zeng, B. Liu, D. Fu, and J. Fu, “Pixel-bert: Aligning image pixels with text by deep multi-modal transformers,”ArXiv preprint, vol. abs/2004.00849, 2020

    2004 arXiv

  116. [124]

    Seeing out of the box: End-to-end pre-training for vision-language representation learning,

    Z. Huang, Z. Zeng, Y . Huang, B. Liu, D. Fu, and J. Fu, “Seeing out of the box: End-to-end pre-training for vision-language representation learning,” inProc. of CVPR, 2021, pp. 12 976–12 985

    2021

  117. [125]

    KD- VLP: Improving end-to-end vision-and-language pretraining with ob- ject knowledge distillation,

    Y . Liu, C. Wu, S.-Y . Tseng, V . Lal, X. He, and N. Duan, “KD- VLP: Improving end-to-end vision-and-language pretraining with ob- ject knowledge distillation,” inProc. of ACL Findings, 2022, pp. 1589– 1600

    2022

  118. [126]

    Xgpt: Cross-modal generative pre-training for image captioning,

    Q. Xia, H. Huang, N. Duan, D. Zhang, L. Ji, Z. Sui, E. Cui, T. Bharti, and M. Zhou, “Xgpt: Cross-modal generative pre-training for image captioning,” inProc. of NLPCC, 2021

    2021

  119. [127]

    M3P: learning universal representations via multitask multilingual multimodal pre-training,

    M. Ni, H. Huang, L. Su, E. Cui, T. Bharti, L. Wang, D. Zhang, and N. Duan, “M3P: learning universal representations via multitask multilingual multimodal pre-training,” inProc. of CVPR, 2021, pp. 3977–3986

    2021

  120. [128]

    Pali: A jointly-scaled multilingual language-image model,

    X. Chen, X. Wang, S. Changpinyo, A. J. Piergiovanni, P. Padlewski, D. Salz, S. Goodman, A. Grycner, B. Mustafa, L. Beyer, A. Kolesnikov, J. Puigcerver, N. Ding, K. Rong, H. Akbari, G. Mishra, L. Xue, A. V . Thapliyal, J. Bradbury, and W. Kuo, “Pali: A jointly-scaled multilingu...

    2023

  121. [129]

    Git: A generative image-to-text transformer for vision and language,

    J. Wang, Z. Yang, X. Hu, L. Li, K. Lin, Z. Gan, Z. Liu, C. Liu, and L. Wang, “Git: A generative image-to-text transformer for vision and language,”ArXiv preprint, vol. abs/2205.14100, 2022

    2022 arXiv

  122. [130]

    Faster R-CNN: towards real-time object detection with region proposal networks,

    S. Ren, K. He, R. B. Girshick, and J. Sun, “Faster R-CNN: towards real-time object detection with region proposal networks,” inProc. of NeurIPS, 2015, pp. 91–99

    2015

  123. [131]

    Deep residual learning for image recognition,

    K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” inProc. of CVPR, 2016, pp. 770–778

    2016

  124. [132]

    Distilled dual-encoder model for vision-language understanding,

    Z. Wang, W. Wang, H. Zhu, M. Liu, B. Qin, and F. Wei, “Distilled dual-encoder model for vision-language understanding,” inProc. of EMNLP, 2022, pp. 8901–8913

    2022

  125. [133]

    LXMERT: Learning cross-modality encoder representations from transformers,

    H. Tan and M. Bansal, “LXMERT: Learning cross-modality encoder representations from transformers,” inProc. of EMNLP, 2019, pp. 5100–5111

    2019

  126. [134]

    MDETR - modulated detection for end-to-end multi-modal under- standing,

    A. Kamath, M. Singh, Y . LeCun, G. Synnaeve, I. Misra, and N. Carion, “MDETR - modulated detection for end-to-end multi-modal under- standing,” inProc. of ICCV, 2021, pp. 1760–1770

    2021

  127. [135]

    Attention bottlenecks for multimodal fusion,

    A. Nagrani, S. Yang, A. Arnab, A. Jansen, C. Schmid, and C. Sun, “Attention bottlenecks for multimodal fusion,” inProc. of NeurIPS, 2021, pp. 14 200–14 213

    2021

  128. [136]

    mPLUG: Effective and efficient vision-language learning by cross-modal skip- connections,

    C. Li, H. Xu, J. Tian, W. Wang, M. Yan, B. Bi, J. Ye, H. Chen, G. Xu, Z. Cao, J. Zhang, S. Huang, F. Huang, J. Zhou, and L. Si, “mPLUG: Effective and efficient vision-language learning by cross-modal skip- connections,” inProc. of EMNLP, 2022, pp. 7241–7259

    2022

  129. [137]

    A survey of vision-language pre-trained models,

    Y . Du, Z. Liu, J. Li, and W. X. Zhao, “A survey of vision-language pre-trained models,” inProc. of IJCAI, 2022, pp. 5436–5443

    2022

  130. [138]

    Decoupling the role of data, attention, and losses in multimodal transformers,

    L. A. Hendricks, J. Mellor, R. Schneider, J.-B. Alayrac, and A. Ne- matzadeh, “Decoupling the role of data, attention, and losses in multimodal transformers,”Transactions of the Association for Com- putational Linguistics, vol. 9, pp. 570–585, 2021

    2021

  131. [139]

    Large-scale adversarial training for vision-and-language representation learning,

    Z. Gan, Y . Chen, L. Li, C. Zhu, Y . Cheng, and J. Liu, “Large-scale adversarial training for vision-and-language representation learning,” in Proc. of NeurIPS, 2020

    2020

  132. [140]

    Deep visual-semantic alignments for generating image descriptions,

    A. Karpathy and F. Li, “Deep visual-semantic alignments for generating image descriptions,” inProc. of CVPR, 2015, pp. 3128–3137

    2015

  133. [141]

    Tips and tricks for visual question answering: Learnings from the 2017 challenge,

    D. Teney, P. Anderson, X. He, and A. van den Hengel, “Tips and tricks for visual question answering: Learnings from the 2017 challenge,” in Proc. of CVPR, 2018, pp. 4223–4232

    2017

  134. [142]

    Randaugment: Practical automated data augmentation with a reduced search space,

    E. D. Cubuk, B. Zoph, J. Shlens, and Q. Le, “Randaugment: Practical automated data augmentation with a reduced search space,” inProc. of NeurIPS, 2020. IEEE TRANSACTIONS ON CIRCUITS AND SYSTEMS FOR VIDEO TECHNOLOGY 31

    2020

  135. [143]

    Mm1. 5: Methods, analysis & insights from multimodal llm fine-tuning,

    H. Zhang, M. Gao, Z. Gan, P. Dufter, N. Wenzel, F. Huang, D. Shah, X. Du, B. Zhang, Y . Liet al., “Mm1. 5: Methods, analysis & insights from multimodal llm fine-tuning,”ArXiv preprint, vol. abs/2409.20566, 2024. Xiao Xureceived the B.S. degree from North- eastern University, S...

    2024 arXiv

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.