Pith. sign in

REVIEW 3 major objections 2 minor 83 references

Sample-efficient Integration of New Modalities into Large Language Models

T0 review · 3 major / 2 minor · reviewed 2026-08-05 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A hypernetwork trained on images, audio, and video can generate a projector that lets a large language model describe unseen modalities—satellite scenes, galaxies, motion sensors, and molecules—starting from as few as 32 paired samples.

desk verdict A genuinely novel hypernetwork approach to few-shot modality integration, with mostly solid experiments, but the abstract's 64x claim is an outlier and should be revised to the paper's own 16x figure. read the letter →

arxiv 2509.04606 v1 pith:ZLKOZMDD submitted 2025-09-04 cs.CL cs.AIcs.CV

Osman Batur İnce , André F. T. Martins , Oisin Mac Aodha , Edoardo M. Ponti This is my paper
classification cs.CLcs.AIcs.CV
keywords multimodallargelanguagemodelssample-efficientadaptationhypernetworkmodalityintegrationlow-resourcemodalitiesLoRAadaptersfew-shotlearninggeneralization
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper sets out to show that a large language model can be extended to a brand-new input modality using only a handful of paired samples. The proposed method, SEMI, trains a hypernetwork on high-resource modalities to generate a low-rank adapter for a shared projector that sits between any modality encoder and the LLM. At test time the hypernetwork sees a few samples from the unseen modality and produces a starting projector, which is then fine-tuned on the same few samples. Across four held-out modalities and three encoder sizes, this beats training the projector from scratch or fine-tuning a shared projector, and the paper reports that matching 32-shot SEMI would require about 64x more data with the from-scratch baseline. The authors posit that the structure of modality-to-text alignment learned from resource-rich modalities is general enough to transfer.

What carries the argument

The load-bearing object is a hypernetwork that generates LoRA adapters for a shared two-layer MLP projector connecting modality encoders to the LLM. The hypernetwork is conditioned on a small interleaved set of modality and text embeddings, anchored by a frozen text encoder; it outputs low-rank updates so that generating all projector weights is replaced by generating (N+M)*R parameters. Training is diversified by applying random Haar-distributed orthogonal matrices to encoder outputs, emulating many new encoders without collecting them, and new encoders with larger embedding dimensions are reduced with Infinite Feature Selection before projection. The generated adapter is averaged over batc

What would settle it

Take a genuinely novel held-out modality, e.g., EEG recordings or 3D point clouds, with an off-the-shelf encoder and a few hundred paired captions. Train SEMI exactly as described, then compare with a from-scratch projector trained on the same 32 samples. If the hypernetwork-generated adapter does not outperform random initialization by a clear margin, the transfer premise fails. A cheaper check: remove one of the three training modalities (say video) and measure whether gains on all four test modalities collapse; the paper predicts they should remain substantial.

Watch

Extended reading notes

Core claim

SEMI's central claim is that adaptation to an unseen modality can be generated rather than learned from scratch. A hypernetwork, trained on image, audio, and video instructions, maps a short interleaved sequence of modality embeddings and text embeddings to LoRA updates for a frozen shared MLP projector. Once merged, these updates give the projector a strong initialization for the new modality, and a brief fine-tuning on the same few samples suffices. On satellite captions, galaxy captions, IMU activity descriptions, and molecule descriptions, SEMI outperforms three baselines at 32 samples and usually remains competitive or best at larger sizes; the strongest baseline, full fine-tuning of th

Load-bearing premise

Learning how to align image, audio, and video embeddings to an LLM is enough to let the hypernetwork generate a useful starting adapter for any other modality, including ones with entirely different input spaces.

Editorial extensions

If this is right

  • Low-resource modalities with only tens of paired samples can be integrated into an 8B LLM rather than waiting for large caption corpora.
  • The same trained hypernetwork is reusable: adding another new modality requires only encoder feature extraction, a short adapter generation, and fine-tuning, not retraining the LLM or the projector.
  • Encoder choices can be flexible; SEMI works when the new encoder's embedding size is smaller or larger than those seen in training.
  • Transfer holds across a similarity spectrum from domain shifts (satellite, galaxies) to entirely unseen input spaces (IMU, molecules), so the benefit is not limited to re-hashing known encoder distributions.
  • At very large data volumes, standard fine-tuning of a shared projector can catch up or surpass SEMI (as on molecules), indicating the advantage is concentrated in the sample-efficiency regime.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The paper's claim that a few rich modalities suffice implies a testable scaling law: performance on held-out modalities should degrade gracefully as training modalities are removed; if it drops sharply with the loss of one modality, the learned structure is less universal than claimed.
  • Because the hypernetwork operates on encoder output distributions rather than raw data, SEMI could in principle integrate any encoder whose embeddings carry signal, including encoders trained without paired text; the limiting factor is the availability of a small paired text set at integration time.
  • The isometric augmentation effectively makes the number of training encoders unbounded; an ablation that adds real diverse encoders while keeping augmentation fixed could reveal whether synthetic diversity substitutes for genuine modal diversity or only delays overfitting.
  • If adapter generation and fine-tuning are decoupled, one could pre-generate adapters for many candidate modalities and select among them during inference, turning modality integration into a retrieval problem.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 2 minor

Summary. The paper proposes SEMI, a three-stage pipeline for integrating new modalities into frozen LLMs. A shared MLP projector is first pre-trained on high-resource image/audio/video captioning data. A hypernetwork is then trained to generate LoRA adapters for this projector, conditioned on a small number of samples (plus instruction and text embeddings) from a modality. At test time, for a previously unseen low-resource modality (satellite images, galaxy images, IMU data, molecules), the hypernetwork generates an adapter from a few samples; the merged projector is then fine-tuned on those same samples. The authors report substantial sample-efficiency gains over three baselines (training a projector from scratch, LoRA adaptation, and full fine-tuning of the pretrained projector), and also introduce a new astronomical captioning dataset (CAPDELS).

Significance. If the central transfer claim holds, SEMI is a practically useful contribution: it offers a way to extend LLMs to new modalities with very little paired data, and it is evaluated across a diverse spectrum of held-out modalities. Strengths of the paper include a systematic comparison across three encoders per image modality, multiple seeds for most datasets, ablations of the main design choices, and the release of code and a new dataset. The main claims are partly overstated, and one part of the training/inference protocol is underspecified, but the core empirical pattern—SEMI helps in low-data regimes—is visible in the reported tables.

major comments (3)
  1. [Abstract and Section 6] The abstract claims 'to reach the same accuracy as 32-shot SEMI, training the projector from scratch needs 64× more data.' This appears to be based on Table 15 (ChEBI-20, Llama 3.1 8B), where 32-shot SEMI BLEU 8.93 is first exceeded by the Projector baseline at 2048 samples (BLEU 10.87). However, this is a single-seed, non-monotonic point: at 8192 samples Projector drops to BLEU 8.42, below the 32-shot SEMI value. The stronger FT Projector baseline reaches comparable BLEU (9.68) already at 512 samples (16×). The paper's own Conclusion and Section 5.1 state 16× ('usually requiring 16× less labelled data'; 'FT Projector requires 16 times more data (512 samples)' for molecules). The 64× claim is an outlier and should be corrected to the supported 16× figure or explicitly qualified as a non-representative best case.
  2. [Section 3.2, Algorithm 2 vs Algorithm 3] During hypernetwork training, a Haar random orthogonal matrix Q is applied to modality embeddings before they are fed to both the hypernetwork and the adapted projector (Algorithm 2, lines 7-12). In Algorithm 3 (few-shot adaptation), no such transformation is applied: raw encoder outputs are used directly. As written, the hypernetwork is trained to generate adapters for Q-transformed inputs but is evaluated on untransformed inputs. This is a potential train-test mismatch and makes the method under-specified. Please clarify whether Q is applied at test time; if not, provide an explicit justification or ablation showing why adapters trained under random rotations transfer to the identity transformation.
  3. [Section 5.1, IMU Data] The text states: 'the best baseline, FT Projector, requires 16 times more examples (2048) to achieve comparable performance to 128-shot SEMI.' Table 14 (SensorCaps) reports a maximum of 1670 samples; there is no 2048 row. At 1670 samples, FT Projector reaches BLEU 15.96 versus 128-shot SEMI's 15.66, a ratio of approximately 13×, not 16×. This claim should be corrected to match the reported data.
minor comments (2)
  1. [Section 5.1, Figure 2] The text refers to 'the encoder with the largest dimensionality, i.e. the ResNet-50 variant (Figure 2b).' Figure 2b is ViT-Large-14; the ResNet-50 results are in Figure 2c. Please fix the cross-reference.
  2. [Abstract vs Section 4] The abstract lists the high-resource training modalities as 'text, speech, audio, video,' but the experiments (Section 4, Table 1) use image, audio, and video with text as the conditioning/output modality. 'Speech' is not used as a separate train modality. Please harmonize the terminology.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the central claim is an empirical generalization result evaluated on held-out test modalities, with fixed, non-tuned hyperparameters.

full rationale

The paper's central claim is that a hypernetwork trained on image/audio/video can generate projectors for unseen low-resource modalities from a few samples. This is an empirical generalization claim, not a derivation from the target data. The target modalities (satellite, astronomical, IMU, molecules) are held out during both projector pre-training and hypernetwork training (Section 1, Table 2), and test metrics are computed on held-out test splits described in Table 2. No target-test values are used to fit the method or to select constants: Appendix K.3 states 'The adaptation hyperparameters are not tuned, generally following the same values except for the learning rate scheduler', and LoRA rank/alpha are fixed at 32/32 for all baselines. Model selection is performed on validation sets ('We perform model selection through early stopping according to the model's CIDEr (or BLEU when unavailable) on the validation sets'), which is standard and does not make the evaluation circular. The abstract's '64x more data' figure is an empirical ratio from Table 15, and the paper itself later summarizes a '16x' gain (Section 6); whether the 64x phrasing is representative is a consistency/correctness concern, not circularity. Self-citations (e.g., [54]) are background references to modular deep learning and are not load-bearing; no uniqueness theorem or ansatz is imported from the authors' prior work. The few-shot fine-tuning stage intentionally reuses the same small training sample that conditions the hypernetwork, but the reported accuracy is measured on separate test sets, so the result does not reduce to its input by construction.

Assumptions & free parameters 5 free parameters · 5 assumptions · 0 invented entities

The method relies on standard deep learning building blocks (MLP projector, LoRA, hypernetwork, attention) and a set of domain assumptions about encoder availability, few-shot paired data, and cross-modal transfer. No new physical entities or forces are introduced. The free parameters are conventional hyperparameters for the architectural components.

free parameters (5)
  • LoRA rank = 32
    Rank of the adapters generated by the hypernetwork; chosen without per-target tuning.
  • LoRA alpha = 32
    Scaling factor for the LoRA update; fixed across all experiments.
  • Hypernetwork context length = 259
    Determines the number of conditioning samples (128 modality-text pairs) fed to the hypernetwork. Larger context was ablated and did not help.
  • Hypernetwork hidden size = 768
    Hidden dimension of the hypernetwork and projector MLP; a design choice not tied to the data.
  • Training hyperparameters = lr=1e-4, various epochs and batch sizes (Appendix K)
    Learning rates, epochs, and batch sizes for projector pre-training, hypernetwork training, and few-shot adaptation; chosen once and not tuned per target modality.
assumptions (5)
  • domain assumption The hypernetwork trained on image, audio, and video captures a universal structure of modality-to-text alignment that transfers to arbitrary unseen modalities.
    Stated in Section 1 as the paper's core hypothesis; without this, the generated adapters would not generalize.
  • domain assumption Pre-trained encoders for the target modalities exist and can be frozen.
    Section 3 assumes a given encoder for each modality; the method does not train encoders.
  • domain assumption A small set of paired modality-text samples from the target modality is available at adaptation time.
    Section 3.3 uses few-shot samples to generate and fine-tune the adapter; without such samples there is no adaptation signal.
  • domain assumption Random orthogonal transformations of encoder outputs emulate new encoders sufficiently well for hypernetwork training.
    Section 3.2 uses Haar-distributed orthogonal matrices to multiply training encoders; this assumes isometric rotations preserve the alignment structure needed for transfer.
  • domain assumption Text embeddings serve as a stable anchor across different modality embedding spaces.
    Section 3.2 grounds modality embeddings by concatenating text embeddings; the ablation in Appendix C suggests this assumption is not uniformly beneficial.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Sample-efficient Integration of New Modalities into Large Language Models." pith.science (2026). https://pith.science/paper/ZLKOZMDD

@misc{pith2026250904606,
  author       = {Pith},
  title        = {Pith review of: Sample-efficient Integration of New Modalities into Large Language Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/ZLKOZMDD}},
  note         = {Machine review of arXiv:2509.04606}
}
abstract

Multimodal foundation models can process several modalities. However, since the space of possible modalities is large and evolving over time, training a model from scratch to encompass all modalities is unfeasible. Moreover, integrating a modality into a pre-existing foundation model currently requires a significant amount of paired data, which is often not available for low-resource modalities. In this paper, we introduce a method for sample-efficient modality integration (SEMI) into Large Language Models (LLMs). To this end, we devise a hypernetwork that can adapt a shared projector -- placed between modality-specific encoders and an LLM -- to any modality. The hypernetwork, trained on high-resource modalities (i.e., text, speech, audio, video), is conditioned on a few samples from any arbitrary modality at inference time to generate a suitable adapter. To increase the diversity of training modalities, we artificially multiply the number of encoders through isometric transformations. We find that SEMI achieves a significant boost in sample efficiency during few-shot integration of new modalities (i.e., satellite images, astronomical images, inertial measurements, and molecules) with encoders of arbitrary embedding dimensionality. For instance, to reach the same accuracy as 32-shot SEMI, training the projector from scratch needs 64$\times$ more data. As a result, SEMI holds promise to extend the modality coverage of foundation models.

Figures

Figures reproduced from arXiv: 2509.04606 by the authors.

Figure 1
Figure 1. High-level framework of SEMI. Left: A hypernetwork is trained to generate an appropriate projector adaptation for high-resource training modalities. Right: A projector is generated by the hypernetwork for any unseen low-resource modality given only a few samples and is then fine-tuned on the same data. This enables the integration of new modalities with minimal training and paired data. Note that some modalities (au… view at source ↗
Figure 2
Figure 2. SydneyCaptions satellite captioning results with three different encoders. The shaded areas [PITH_FULL_IMAGE:figures/full_fig_p007_2.png] view at source ↗
Figure 3
Figure 3. CAPDELS astronomical image captioning results with three different encoders. The shaded [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗
Figures from the paper (7 more)
Figure 4
Figure 4. Figure 4: Left: SensorCaps activity description dataset results. Right: ChEBI-20 molecule descrip￾tion dataset results. 0.0 0.2 0.4 0.6 0.8 1.0 0.0 0.2 0.4 0.6 0.8 1.0 Ours FT Projector Projector LoRA 10 2 10 3 Dataset Size (log-scale) 0 2 4 6 8 BLEU 10 2 10 3 Dataset Size (log-…
Figure 5
Figure 5. Figure 5: SoundBible audio captioning results. Left: Llama 3.2 1B Instruct. Right: Llama 3.1 8B Instruct. our method exhibits the largest gains over baselines when integrating the encoder with the largest dimensionality, i.e. the ResNet-50 variant (Figure 2b). These results high…
Figure 6
Figure 6. Figure 6: The frequency of most frequent label sets before ( [PITH_FULL_IMAGE:figures/full_fig_p015_6.png]
Figure 7
Figure 7. Figure 7: Comparison of different adapter integration techniques for the SydneyCaptions dataset. [PITH_FULL_IMAGE:figures/full_fig_p016_7.png]
Figure 8
Figure 8. Figure 8: Results with PCA and Inf-FS dimensionality reduction techniques for SydneyCaptions [PITH_FULL_IMAGE:figures/full_fig_p017_8.png]
Figure 9
Figure 9. Figure 9: Comparison of single adapter versus averaging multiple adapters for SydneyCaptions - [PITH_FULL_IMAGE:figures/full_fig_p018_9.png]
Figure 10
Figure 10. Figure 10: Linear CKA scores between modality embeddings at different stages and text embeddings. [PITH_FULL_IMAGE:figures/full_fig_p018_10.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

83 extracted references · 57 canonical work pages

  1. [1]

    Flamingo: a Visual Language Model for Few-Shot Learning

    Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, Roman Ring, Eliza Rutherford, Serkan Cabi, Tengda Han, Zhitao Gong, Sina Samangooei, Marianne Monteiro, Jacob L Menick, Sebastian Borgeaud, Andy Brock, Aida Nematzadeh, Sahand Sharifzadeh, Mikoł aj Bi´nk...

    2022

  2. [2]

    METEOR: An Automatic Metric for MT Evaluation with Improved Correlation with Human Judgments

    Satanjeev Banerjee and Alon Lavie. METEOR: An Automatic Metric for MT Evaluation with Improved Correlation with Human Judgments. In Proceedings of the ACL Workshop on Intrinsic and Extrinsic Evaluation Measures for Machine Translation and/or Summarization, pages 65–72, Ann Arbor, Michigan, June 2005. Association for Computational Linguistics

    2005

  3. [3]

    SciBERT: A Pretrained Language Model for Scientific Text

    Iz Beltagy, Kyle Lo, and Arman Cohan. SciBERT: A Pretrained Language Model for Scientific Text. In EMNLP. Association for Computational Linguistics, 2019

    2019

  4. [4]

    NLTK: The Natural Language Toolkit

    Steven Bird and Edward Loper. NLTK: The Natural Language Toolkit. In Proceedings of the ACL Interactive Poster and Demonstration Sessions, pages 214–217, Barcelona, Spain, July

    work page

  5. [5]

    Principled Weight Initialization for Hypernet- works

    Oscar Chang, Lampros Flokas, and Hod Lipson. Principled Weight Initialization for Hypernet- works. In International Conference on Learning Representations, 2020

    work page 2020

  6. [6]

    Vinod Kumar Chauhan, Jiandong Zhou, Ping Lu, Soheila Molaei, and David A. Clifton. A brief review of hypernetworks in deep learning. Artificial Intelligence Review, 57(9):250, Aug 2024

    work page 2024

  7. [7]

    Model Composition for Multimodal Large Language Models

    Chi Chen, Yiyang Du, Zheng Fang, Ziyue Wang, Fuwen Luo, Peng Li, Ming Yan, Ji Zhang, Fei Huang, Maosong Sun, and Yang Liu. Model Composition for Multimodal Large Language Models. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar, editors, Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages...

    work page 2024

  8. [8]

    VisualGPT: Data-efficient Image Captioning by Balancing Visual Input and Linguistic Knowledge from Pretraining.CoRR, abs/2102.10407, 2021

    Jun Chen, Han Guo, Kai Yi, Boyang Li, and Mohamed Elhoseiny. VisualGPT: Data-efficient Image Captioning by Balancing Visual Input and Linguistic Knowledge from Pretraining.CoRR, abs/2102.10407, 2021

    arXiv 2021

Show all 83 references
  1. [9]

    ShareGPT4V dataset on Huggingface, 2024

    Lin Chen. ShareGPT4V dataset on Huggingface, 2024. Accessed on March 6, 2025 in https://huggingface.co/datasets/Lin-Chen/ShareGPT4V, as described in [10]

    2024

  2. [10]

    ShareGPT4V: Improving Large Multi-Modal Models with Better Captions

    Lin Chen, Jisong Li, Xiaoyi Dong, Pan Zhang, Conghui He, Jiaqi Wang, Feng Zhao, and Dahua Lin. ShareGPT4V: Improving Large Multi-Modal Models with Better Captions. arXiv preprint arXiv:2311.12793, 2023

    2023 arXiv

  3. [11]

    ShareGPT4Video: Improving Video Understanding and Generation with Better Captions

    Lin Chen, Xilin Wei, Jinsong Li, Xiaoyi Dong, Pan Zhang, Yuhang Zang, Zehui Chen, Haodong Duan, Bin Lin, Zhenyu Tang, et al. ShareGPT4Video: Improving Video Understanding and Generation with Better Captions. arXiv preprint arXiv:2406.04325, 2024

    2024 arXiv

  4. [12]

    InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuning

    Wenliang Dai, Junnan Li, Dongxu Li, Anthony Tiong, Junqi Zhao, Weisheng Wang, Boyang Li, Pascale N Fung, and Steven Hoi. InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuning. In Advances in Neural Information Processing Systems, 2023

    2023

  5. [13]

    Clotho: an Audio Captioning Dataset

    Konstantinos Drossos, Samuel Lipping, and Tuomas Virtanen. Clotho: an Audio Captioning Dataset. ICASSP 2020 - 2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 736–740, 2019

    2020

  6. [14]

    Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, Anirudh Goyal, An- thony S. Hartshorn, Aobo Yang, Archi Mitra, Archie Sravankumar, Artem Korenev, Arthur Hinsvark, Arun Rao, Asto...

    2024 arXiv

  7. [15]

    Translation between Molecules and Natural Language

    Carl Edwards, Tuan Lai, Kevin Ros, Garrett Honke, Kyunghyun Cho, and Heng Ji. Translation between Molecules and Natural Language. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 375–413, Abu Dhabi, United Arab Emirates, Dec. 202...

    2022

  8. [16]

    Text2Mol: Cross-Modal Molecule Retrieval with Natural Language Queries

    Carl Edwards, ChengXiang Zhai, and Heng Ji. Text2Mol: Cross-Modal Molecule Retrieval with Natural Language Queries. In Marie-Francine Moens, Xuanjing Huang, Lucia Specia, and Scott Wen-tau Yih, editors,Proceedings of the 2021 Conference on Empirical Methods in Natural Language...

    2021

  9. [17]

    CLAP: Learning Audio Concepts From Natural Language Supervision

    Benjamin Elizalde, Soham Deshmukh, Mahmoud Al Ismail, and Huaming Wang. CLAP: Learning Audio Concepts From Natural Language Supervision. In ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 1–5. IEEE, 2023

    2023

  10. [18]

    LLaMA-Adapter V2: Parameter- Efficient Visual Instruction Model

    Peng Gao, Jiaming Han, Renrui Zhang, Ziyi Lin, Shijie Geng, Aojun Zhou, Wei Zhang, Pan Lu, Conghui He, Xiangyu Yue, Hongsheng Li, and Yu Qiao. LLaMA-Adapter V2: Parameter- Efficient Visual Instruction Model. arXiv preprint arXiv:2304.15010, 2023

    2023 arXiv

  11. [19]

    Making LLaMA SEE and Draw with SEED Tokenizer

    Yuying Ge, Sijie Zhao, Ziyun Zeng, Yixiao Ge, Chen Li, Xintao Wang, and Ying Shan. Making LLaMA SEE and Draw with SEED Tokenizer. arXiv preprint arXiv:2310.01218, 2023

    2023 arXiv

  12. [20]

    EMMA: Efficient Visual Alignment in Multi-Modal LLMs

    Sara Ghazanfari, Alexandre Araujo, Prashanth Krishnamurthy, Siddharth Garg, and Farshad Khorrami. EMMA: Efficient Visual Alignment in Multi-Modal LLMs. ArXiv, abs/2410.02080, 2024

    2024 arXiv

  13. [21]

    Agent Smith: A Single Image Can Jailbreak One Million Multimodal LLM Agents Exponentially Fast

    Xiangming Gu, Xiaosen Zheng, Tianyu Pang, Chao Du, Qian Liu, Ye Wang, Jing Jiang, and Min Lin. Agent Smith: A Single Image Can Jailbreak One Million Multimodal LLM Agents Exponentially Fast. In ICML, 2024

    2024

  14. [22]

    LLaV A-UHD: an LMM Perceiving Any Aspect Ratio and High-Resolution Images

    Zonghao Guo, Ruyi Xu, Yuan Yao, Junbo Cui, Zanlin Ni, Chunjiang Ge, Tat-Seng Chua, Zhiyuan Liu, and Gao Huang. LLaV A-UHD: an LMM Perceiving Any Aspect Ratio and High-Resolution Images. In ECCV, 2024

    2024

  15. [23]

    Dai, and Quoc V

    David Ha, Andrew M. Dai, and Quoc V . Le. HyperNetworks. InInternational Conference on Learning Representations, 2017

    2017

  16. [24]

    OneLLM: One Framework to Align All Modalities with Language

    Jiaming Han, Kaixiong Gong, Yiyuan Zhang, Jiaqi Wang, Kaipeng Zhang, Dahua Lin, Yu Qiao, Peng Gao, and Xiangyu Yue. OneLLM: One Framework to Align All Modalities with Language. CVPR, pages 26574–26585, 2023

    2023

  17. [25]

    ImageBind-LLM: Multi-modality Instruction Tuning

    Jiaming Han, Renrui Zhang, Wenqi Shao, Peng Gao, Peng Xu, Han Xiao, Kaipeng Zhang, Chris Liu, Song Wen, Ziyu Guo, Xudong Lu, Shuai Ren, Yafei Wen, Xiaoxin Chen, Xiangyu Yue, Hongsheng Li, and Yu Jiao Qiao. ImageBind-LLM: Multi-modality Instruction Tuning. ArXiv, abs/2309.03905, 2023

    2023 arXiv

  18. [26]

    LoRA: Low-Rank Adaptation of Large Language Models

    Edward J Hu, yelong shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. LoRA: Low-Rank Adaptation of Large Language Models. In International Conference on Learning Representations, 2022

    2022

  19. [27]

    LLaSA: A Multimodal LLM for Human Activity Analysis Through Wearable and Smartphone Sensors, 2025

    Sheikh Asif Imran, Mohammad Nur Hossain Khan, Subrata Biswas, and Bashima Islam. LLaSA: A Multimodal LLM for Human Activity Analysis Through Wearable and Smartphone Sensors, 2025

    2025

  20. [28]

    Perceiver: General perception with iterative attention

    Andrew Jaegle, Felix Gimeno, Andy Brock, Oriol Vinyals, Andrew Zisserman, and Joao Carreira. Perceiver: General perception with iterative attention. In International Conference on Machine Learning, pages 4651–4664. PMLR, 2021

    2021

  21. [29]

    From Specific-MLLM to Omni-MLLM: A Survey about the MLLMs alligned with Multi-Modality

    Shixin Jiang, Jiafeng Liang, Ming Liu, and Bing Qin. From Specific-MLLM to Omni-MLLM: A Survey about the MLLMs alligned with Multi-Modality. arXiv preprint arXiv:2412.11694, 2024

    2024 arXiv

  22. [30]

    BRA VE: Broadening the visual encoding of vision-language models

    O˘guzhan Fatih Kar, Alessio Tonioni, Petra Poklukar, Achin Kulshrestha, Amir Zamir, and Federico Tombari. BRA VE: Broadening the visual encoding of vision-language models. In European Conference on Computer Vision, pages 113–132. Springer, 2024

    2024

  23. [31]

    AudioCaps: Generat- ing Captions for Audios in The Wild

    Chris Dongjoo Kim, Byeongchang Kim, Hyunmin Lee, and Gunhee Kim. AudioCaps: Generat- ing Captions for Audios in The Wild. In NAACL-HLT, 2019

    2019

  24. [32]

    Berg, Wan-Yen Lo, Piotr Dollár, and Ross Girshick

    Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C. Berg, Wan-Yen Lo, Piotr Dollár, and Ross Girshick. Segment Anything. In ICCV, pages 3992–4003, 2023

    2023

  25. [33]

    Grounding Language Models to Images for Multimodal Inputs and Outputs

    Jing Yu Koh, Ruslan Salakhutdinov, and Daniel Fried. Grounding Language Models to Images for Multimodal Inputs and Outputs. ICML, 2023. 11

    2023

  26. [34]

    Similarity of Neural Network Representations Revisited

    Simon Kornblith, Mohammad Norouzi, Honglak Lee, and Geoffrey Hinton. Similarity of Neural Network Representations Revisited. In Kamalika Chaudhuri and Ruslan Salakhutdinov, editors, Proceedings of the 36th International Conference on Machine Learning, volume 97 of Proceedings ...

    2019

  27. [35]

    BLIP-2: Bootstrapping Language- Image Pre-training with Frozen Image Encoders and Large Language Models

    Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. BLIP-2: Bootstrapping Language- Image Pre-training with Frozen Image Encoders and Large Language Models. In ICML, 2023

    2023

  28. [36]

    ROUGE: A Package for Automatic Evaluation of Summaries

    Chin-Yew Lin. ROUGE: A Package for Automatic Evaluation of Summaries. InText Summariza- tion Branches Out, pages 74–81, Barcelona, Spain, July 2004. Association for Computational Linguistics

    2004

  29. [37]

    Microsoft COCO: Common Objects in Context

    Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft COCO: Common Objects in Context. In Computer Vision–ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings...

    2014

  30. [38]

    RemoteCLIP: A Vision Language Foundation Model for Remote Sensing

    Fan Liu, Delong Chen, Zhangqingyun Guan, Xiaocong Zhou, Jiale Zhu, Qiaolin Ye, Liyong Fu, and Jun Zhou. RemoteCLIP: A Vision Language Foundation Model for Remote Sensing. IEEE Transactions on Geoscience and Remote Sensing, 62:1–16, 2024

    2024

  31. [39]

    Visual Instruction Tuning

    Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual Instruction Tuning. In A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine, editors,Advances in Neural Information Processing Systems, 2023

    2023

  32. [40]

    Towards Modality Generalization: A Benchmark and Prospective Analysis

    Xiaohao Liu, Xiaobo Xia, Zhuo Huang, and Tat-Seng Chua. Towards Modality Generalization: A Benchmark and Prospective Analysis. arXiv preprint arXiv:2412.18277, 2024

    2024 arXiv

  33. [41]

    MolCA: Molecular Graph-Language Modeling with Cross-Modal Projector and Uni-Modal Adapter

    Zhiyuan Liu, Sihang Li, Yanchen Luo, Hao Fei, Yixin Cao, Kenji Kawaguchi, Xiang Wang, and Tat-Seng Chua. MolCA: Molecular Graph-Language Modeling with Cross-Modal Projector and Uni-Modal Adapter. In Houda Bouamor, Juan Pino, and Kalika Bali, editors,Proceedings of the 2023 Con...

    2023

  34. [42]

    Decoupled Weight Decay Regularization

    Ilya Loshchilov and Frank Hutter. Decoupled Weight Decay Regularization. In International Conference on Learning Representations, 2019

    2019

  35. [43]

    Unified-IO 2: Scaling Autoregressive Multimodal Models with Vision Language Audio and Action

    Jiasen Lu, Christopher Clark, Sangho Lee, Zichen Zhang, Savya Khosla, Ryan Marten, Derek Hoiem, and Aniruddha Kembhavi. Unified-IO 2: Scaling Autoregressive Multimodal Models with Vision Language Audio and Action. In CVPR, pages 26439–26455, June 2024

    2024

  36. [44]

    Macaw-LLM: Multi-Modal Language Modeling with Image, Audio, Video, and Text Integration

    Chenyang Lyu, Minghao Wu, Longyue Wang, Xinting Huang, Bingshuai Liu, Zefeng Du, Shuming Shi, and Zhaopeng Tu. Macaw-LLM: Multi-Modal Language Modeling with Image, Audio, Video, and Text Integration. arXiv preprint arXiv:2306.09093, 2023

    2023 arXiv

  37. [45]

    Ee-mllm: A data-efficient and compute-efficient multi- modal large language model

    Feipeng Ma, Yizhou Zhou, Zheyu Zhang, Shilin Yan, Hebei Li, Zilong He, Siying Wu, Fengyun Rao, Yueyi Zhang, and Xiaoyan Sun. Ee-mllm: A data-efficient and compute-efficient multi- modal large language model. arXiv preprint arXiv:2408.11795, 2024

    2024 arXiv

  38. [46]

    Plumbley, Yuexian Zou, and Wenwu Wang

    Xinhao Mei, Chutong Meng, Haohe Liu, Qiuqiang Kong, Tom Ko, Chengqi Zhao, Mark D. Plumbley, Yuexian Zou, and Wenwu Wang. WavCaps: A ChatGPT-Assisted Weakly-Labelled Audio Captioning Dataset for Audio-Language Multimodal Research. IEEE/ACM Trans. Audio, Speech and Lang. Proc., ...

    2024

  39. [47]

    Llama 3.2: Model Cards and Prompt formats, 2024

    Meta. Llama 3.2: Model Cards and Prompt formats, 2024. Accessed on February 28, 2025 in https://www.llama.com/docs/model-cards-and-prompt-formats/ llama3_2/, a follow-up study on [14]

    2024

  40. [48]

    How to generate random matrices from the classical compact groups

    Francesco Mezzadri. How to generate random matrices from the classical compact groups. Notices of the American Mathematical Society, 54(5):592–604, 2007

    2007

  41. [49]

    Clipcap: Clip prefix for image captioning

    Ron Mokady, Amir Hertz, and Amit H Bermano. Clipcap: Clip prefix for image captioning. arXiv preprint arXiv:2111.09734, 2021

    2021 arXiv

  42. [50]

    AnyMAL: An Efficient and Scalable Any-Modality Augmented Language Model

    Seungwhan Moon, Andrea Madotto, Zhaojiang Lin, Tushar Nagarajan, Matt Smith, Shashank Jain, Chun-Fu Yeh, Prakash Murugesan, Peyman Heidari, Yue Liu, Kavya Srinet, Babak Damavandi, and Anuj Kumar. AnyMAL: An Efficient and Scalable Any-Modality Augmented Language Model. In Franc...

    2024

  43. [51]

    OpenVid dataset on Huggingface, 2025

    Kepan Nan. OpenVid dataset on Huggingface, 2025. Accessed on March 5, 2025 in https: //huggingface.co/datasets/nkp37/OpenVid-1M, as described in [52]. 12

    2025

  44. [52]

    Openvid-1m: A large-scale high-quality dataset for text-to-video generation

    Kepan Nan, Rui Xie, Penghao Zhou, Tiehan Fan, Zhenheng Yang, Zhijie Chen, Xiang Li, Jian Yang, and Ying Tai. Openvid-1m: A large-scale high-quality dataset for text-to-video generation. arXiv preprint arXiv:2407.02371, 2024

    2024 arXiv

  45. [53]

    Bleu: a method for automatic evaluation of machine translation

    Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. Bleu: a method for automatic evaluation of machine translation. In ACL, pages 311–318, 2002

    2002

  46. [54]

    Modular Deep Learning

    Jonas Pfeiffer, Sebastian Ruder, Ivan Vuli ´c, and Edoardo Ponti. Modular Deep Learning. Transactions on Machine Learning Research, 2023. Survey Certification

    2023

  47. [55]

    Deep semantic understanding of high resolution remote sensing image

    Bo Qu, Xuelong Li, Dacheng Tao, and Xiaoqiang Lu. Deep semantic understanding of high resolution remote sensing image. In 2016 International Conference on Computer, Information and Telecommunication Systems (CITS), pages 1–5, 2016

    2016

  48. [56]

    Learning Transferable Visual Models From Natural Language Supervision

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agar- wal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning Transferable Visual Models From Natural Language Supervision. In Marina Meila an...

    2021

  49. [57]

    Infinite Feature Selection

    Giorgio Roffo, Simone Melzi, and Marco Cristani. Infinite Feature Selection. In 2015 IEEE International Conference on Computer Vision (ICCV), pages 4202–4210, 2015

    2015

  50. [58]

    Learning to Control Fast-Weight Memories: An Alternative to Dynamic Recurrent Networks

    Jürgen Schmidhuber. Learning to Control Fast-Weight Memories: An Alternative to Dynamic Recurrent Networks. Neural Computation, 4(1):131–139, 1992

    1992

  51. [59]

    UnIV AL: Unified Model for Image, Video, Audio and Language Tasks

    Mustafa Shukor, Corentin Dancette, Alexandre Rame, and Matthieu Cord. UnIV AL: Unified Model for Image, Video, Audio and Language Tasks. Transactions on Machine Learning Research, 2023

    2023

  52. [60]

    B. D. Simmons, Chris Lintott, Kyle W. Willett, Karen L. Masters, Jeyhan S. Kartaltepe, Boris Häußler, Sugata Kaviraj, Coleman Krawczyk, S. J. Kruk, Daniel H. McIntosh, R. J. Smethurst, Robert C. Nichol, Claudia Scarlata, Kevin Schawinski, Christopher J. Conselice, Omar Almaini...

    2016

  53. [61]

    Siglip 2: Multilingual vision-language encoders with improved semantic understanding, localization, and dense features

    Michael Tschannen, Alexey Gritsenko, Xiao Wang, Muhammad Ferjad Naeem, Ibrahim Alab- dulmohsin, Nikhil Parthasarathy, Talfan Evans, Lucas Beyer, Ye Xia, Basil Mustafa, et al. Siglip 2: Multilingual vision-language encoders with improved semantic understanding, localization, an...

    2025 arXiv

  54. [62]

    Lawrence Zitnick, and Devi Parikh

    Ramakrishna Vedantam, C. Lawrence Zitnick, and Devi Parikh. CIDEr: Consensus-based image description evaluation. In 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 4566–4575, 2015

    2015

  55. [63]

    Oliphant, Matt Haberland, Tyler Reddy, David Cournapeau, Evgeni Burovski, Pearu Peterson, Warren Weckesser, Jonathan Bright, Stéfan J

    Pauli Virtanen, Ralf Gommers, Travis E. Oliphant, Matt Haberland, Tyler Reddy, David Cournapeau, Evgeni Burovski, Pearu Peterson, Warren Weckesser, Jonathan Bright, Stéfan J. van der Walt, Matthew Brett, Joshua Wilson, K. Jarrod Millman, Nikolay Mayorov, Andrew R. J. Nelson, E...

    2020

  56. [64]

    Lintott, Anna M

    Mike Walmsley, Campbell Allen, Ben Aussel, Micah Bowles, Kasia Gregorowicz, Inigo Val Slijepcevic, Chris J. Lintott, Anna M. m. Scaife, Maja Jabło ´nska, Kosio Karchev, Denise Lanzieri, Devina Mohan, David O’Ryan, Bharath Saiguhan, Crisel Suárez, Nicolás Guerra- Varas, and Ren...

    2023

  57. [65]

    VideoCLIP-XL: Advancing Long Description Understanding for Video CLIP Models

    Jiapeng Wang, Chengyu Wang, Kunzhe Huang, Jun Huang, and Lianwen Jin. VideoCLIP-XL: Advancing Long Description Understanding for Video CLIP Models. In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen, editors, Proceedings of the 2024 Conference on Empirical Methods in Natural...

    2024

  58. [66]

    InternVid: A Large-scale Video-Text Dataset for Multimodal Understanding and Generation

    Yi Wang, Yinan He, Yizhuo Li, Kunchang Li, Jiashuo Yu, Xin Ma, Xinhao Li, Guo Chen, Xinyuan Chen, Yaohui Wang, Ping Luo, Ziwei Liu, Yali Wang, Limin Wang, and Yu Qiao. InternVid: A Large-scale Video-Text Dataset for Multimodal Understanding and Generation. In ICLR, 2024. 13

    2024

  59. [67]

    Smarter, better, faster, longer: A modern bidirectional encoder for fast, memory efficient, and long context finetuning and inference

    Benjamin Warner, Antoine Chaffin, Benjamin Clavié, Orion Weller, Oskar Hallström, Said Taghadouini, Alexis Gallagher, Raja Biswas, Faisal Ladhak, Tom Aarsen, et al. Smarter, better, faster, longer: A modern bidirectional encoder for fast, memory efficient, and long context fin...

    2024 arXiv

  60. [68]

    Transformers: State-of-the-Art Natural Language Processing

    Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Remi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mari...

    2020

  61. [69]

    Limu-bert: Unleashing the potential of unlabeled data for imu sensing applications

    Huatao Xu, Pengfei Zhou, Rui Tan, Mo Li, and Guobin Shen. Limu-bert: Unleashing the potential of unlabeled data for imu sensing applications. In Proceedings of the 19th ACM Conference on Embedded Networked Sensor Systems, pages 220–233, 2021

    2021

  62. [70]

    Qwen2.5-Omni Technical Report

    Jin Xu, Zhifang Guo, Jinzheng He, Hangrui Hu, Ting He, Shuai Bai, Keqin Chen, Jialin Wang, Yang Fan, Kai Dang, Bin Zhang, Xiong Wang, Yunfei Chu, and Junyang Lin. Qwen2.5-Omni Technical Report. arXiv preprint arXiv:2503.20215, 2025

    2025 arXiv

  63. [71]

    Xuenan Xu, Zhiling Zhang, Zelin Zhou, Pingyue Zhang, Zeyu Xie, Mengyue Wu, and Kenny Q. Zhu. BLAT: Bootstrapping Language-Audio Pre-training based on AudioSet Tag-guided Synthetic Data. In Proceedings of the 31st ACM International Conference on Multimedia, MM ’23, page 2756–27...

    2023

  64. [72]

    Qwen2.5 Technical Report

    An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jingren Zhou, Junyang Lin, Kai Dang, Keming Lu, Keqin Bao, Kexin Yang, Le Yu, Me...

    2024 arXiv

  65. [73]

    LLMs Can Evolve Continually on Modality for X-Modal Reasoning

    Jiazuo Yu, Haomiao Xiong, Lu Zhang, Haiwen Diao, Yunzhi Zhuge, Lanqing Hong, Dong Wang, Huchuan Lu, You He, and Long Chen. LLMs Can Evolve Continually on Modality for X-Modal Reasoning. ArXiv, abs/2410.20178, 2024

    2024 arXiv

  66. [74]

    How Johnny Can Persuade LLMs to Jailbreak Them: Rethinking Persuasion to Challenge AI Safety by Humanizing LLMs

    Yi Zeng, Hongpeng Lin, Jingwen Zhang, Diyi Yang, Ruoxi Jia, and Weiyan Shi. How Johnny Can Persuade LLMs to Jailbreak Them: Rethinking Persuasion to Challenge AI Safety by Humanizing LLMs. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar, editors, Proceedings of the 62nd Annua...

    2024

  67. [75]

    mGTE: Generalized Long-Context Text Repre- sentation and Reranking Models for Multilingual Text Retrieval

    Xin Zhang, Yanzhao Zhang, Dingkun Long, Wen Xie, Ziqi Dai, Jialong Tang, Huan Lin, Baosong Yang, Pengjun Xie, Fei Huang, et al. mGTE: Generalized Long-Context Text Repre- sentation and Reranking Models for Multilingual Text Retrieval. In Proceedings of the 2024 Conference on E...

    2024

  68. [76]

    BuboGPT: Enabling Visual Grounding in Multi-Modal LLMs

    Yang Zhao, Zhijie Lin, Daquan Zhou, Zilong Huang, Jiashi Feng, and Bingyi Kang. BuboGPT: Enabling Visual Grounding in Multi-Modal LLMs. arXiv preprint arXiv:2307.08581, 2023

    2023 arXiv

  69. [77]

    ChatBridge: Bridging Modalities with Large Language Model as a Language Catalyst

    Zijia Zhao, Longteng Guo, Tongtian Yue, Si-Qing Chen, Shuai Shao, Xinxin Zhu, Zehuan Yuan, and Jing Liu. ChatBridge: Bridging Modalities with Large Language Model as a Language Catalyst. ArXiv, abs/2305.16103, 2023

    2023 arXiv

  70. [78]

    MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Models

    Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny. MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Models. arXiv preprint arXiv:2304.10592, 2023

    2023 arXiv

  71. [79]

    Is the galaxy simply smooth and rounded, with no sign of a disk?

    Ge Zhu, Jordan Darefsky, and Zhiyao Duan. Cacophony: An Improved Contrastive Audio-Text Model. IEEE/ACM Trans. Audio, Speech and Lang. Proc., 32:4867–4879, Oct. 2024. 14 Appendix A CAPDELS dataset 0 5 10 15 20 Morphological Label Set Index 0 500 1000 1500 2000 2500 3000Count 0...

    2024

  72. [82]

    The positive and negative values in both sensor data indicate changes in acceleration and angular velocity dur- ing the walking activity. 4. The peaks and troughs in the data suggest steps taken in a rhythmic pattern. Ours: In the gyroscope data, there are varying values of x,...

  73. [83]

    The data is relatively stable, with slight variations, suggesting a stationary position

    Gyroscope data: - The x, y, and z axes show minimal changes, indicating no significant rotation or movement in these planes. The data is relatively stable, with slight variations, suggesting a stationary position

  74. [84]

    Accelerometer data: - The x, y, and z axes show consistent changes, typically ranging from -1 to 1 g. This 21 Table 7: Qualitative examples for ChEBI-20 dataset and MolCA for methods trained with 2048 samples Ground Truth Predictions 2D structure image: Description: The molecu...

  75. [2024]

    Association for Computational Linguistics

Pith tools

Reviewed August 5, 2026 · model on record in the stance chip above.