Pith. sign in

REVIEW 4 major objections 7 minor 51 references

Mixture of Probes: Learning from Privileged Modalities in Multimodal LLMs Through Probing

T0 review · 4 major / 7 minor · reviewed 2026-07-13 · grok-4.5

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Auxiliary training modalities improve single-modality MLLM inference when probes separate specific from shared signals.

desk verdict Solid privileged-modality recipe for MLLMs; gains are real vs their baselines, but the missing unimodal-MoP control leaves transfer vs better adapter partly confounded. read the letter →

arxiv 2607.08839 v1 pith:UMFTMXB7 submitted 2026-07-09 cs.CV cs.LG

Dominick Reilly , Qiyu Wu , Hiromi Wakaki , Srijan Das , Yuki Mistufuji This is my paper · ORCID
classification cs.CVcs.LG
keywords privilegedmodalitiesmultimodallargelanguagemodelsmixtureofprobesmodality-specificmodality-generalcross-modaltransferprobedisentanglementuniversalmodalityencoder
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Many real deployments of multimodal large language models can use rich sensors only while training, then must answer from one cheap or reliable stream at test time. Standard MLLMs treat every modality as just another input to align into language space, so the extra training streams often fail to help—and can even hurt—the modality that remains at inference. This paper argues that the missing ingredient is deliberate separation of modality-specific structure from modality-general structure inside a shared encoder. It introduces Mixture of Probes (MoP): learnable tokens that cross-attend to intermediate layers of a frozen universal encoder, some probes seeing only one modality and others shared across all. A companion training recipe (MoP-X) keeps those two probe families from collapsing into each other and interleaves modalities inside each batch so the shared probes accumulate transferable signal. Across activities-of-daily-living and music-understanding benchmarks, the resulting single-modality models consistently beat both unimodal training and naive joint multimodal training, showing that privileged modalities can raise the ceiling of the modalities that remain.

What carries the argument

Mixture of Probes (MoP): modality-specific and modality-general learnable tokens that cross-attend layer-wise to a frozen universal encoder, trained with a probe-disentanglement loss (MoP-X) that penalizes similarity among probes so they do not collapse.

What would settle it

Train MoP and the same naive multimodal baseline on the identical multi-modal corpus; if single-modality test accuracy on each held-out stream fails to beat both the unimodal model and the naive joint model on Ego-in-Exo and Music-AVQA, the central claim fails.

Watch

Extended reading notes

Core claim

When auxiliary modalities are available only at training time, an MLLM can still improve its single-modality inference performance by extracting and disentangling modality-specific and modality-general signals from intermediate layers of a shared encoder via a structured mixture of probes, rather than relying solely on final-layer alignment.

Load-bearing premise

That a single frozen shared encoder plus probes updated on one modality per sample (never jointly) is enough to extract transferable cross-modal structure.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 7 minor

Summary. The paper formalizes the privileged-modality setting for MLLMs—auxiliary modalities available only at training—and proposes Mixture of Probes (MoP): learnable modality-specific and modality-general probe tokens that cross-attend to intermediate layers of a frozen shared universal encoder (SigLIP), plus MoP-X (probe disentanglement loss L_dis on off-diagonal cosine similarities and modality-interleaved single-modality batches). Probe tokens are projected into the LLM (Qwen 2.5 + LoRA) alongside final encoder features. On Ego-in-Exo Perception (ego/exo/depth) and Music-AVQA (audio/video), under single-modality inference, MoP outperforms unimodal MLLMs, a naive joint multimodal baseline, X-InstructBLIP, and OneLLM, with ablations on probe types, λ, probing depth, loss placement, probe variance/subspace geometry, and data scale.

Significance. The privileged-modality setting is practically important (e.g., ADL and sensor-asymmetric deployment) and under-addressed in current MLLM alignment work. If the gains truly come from structured use of training-only auxiliaries rather than from probing as a stronger unimodal adapter, the result is a clear advance over final-layer alignment methods. Strengths include a coherent architecture (Eqs. 6–12), multi-domain evaluation with a protocol that isolates each inference modality, diagnostic analyses (Figs. 6–8, Table 3), and a commitment to release code, checkpoints, and protocols. The contribution is empirical and architectural rather than theoretical; its impact hinges on cleanly attributing improvements to privileged transfer.

major comments (4)
  1. [§5.1–5.2, Table 1, Table 3] Central claim attribution is confounded. Abstract, §1, §5.2, and the conclusion state that auxiliary modalities yield substantial single-modality gains when leveraged by MoP. Table 1 compares MoP (probes + multi-modality MoP-X training) to Unimodal MLLM [47] (plain base architecture, target modality only). There is no unimodal-MoP control (same probes, L_dis, and training recipe on the target modality alone). Absolute lifts (e.g., ego 79.46 vs 74.57, exo 74.18 vs 65.68, depth 66.21 vs 57.06) can therefore be explained by intermediate probing as a stronger adapter rather than by cross-modal transfer through shared general probes. Naive multi-training without probes is weaker than MoP, but that only shows probes help under multi-training. Table 3 and Figs. 3–5 ablate probe families and L_dis only inside the multi-trained regime. A unimodal-MoP cell (and ideally multi-trained MoP without ge
  2. [Table 1, §5.1 Compared Models, Abstract] Baseline reliability on Ego-in-Exo is uneven and weakens the “strong MLLM baselines” comparison. OneLLM averages ~40–45% across inference modalities (Table 1), far below Unimodal MLLM (~57–75%) and even random-ish on several tasks (e.g., Action Und. exo 23.17). X-InstructBLIP is closer to Naive MLLM but does not consistently beat Unimodal. Without implementation details, hyperparameter matching, or discussion of why a published unified encoder collapses here, the large relative gains vs OneLLM (and the abstract’s “up to 65% relative improvement”) overstate the method’s advantage relative to competent privileged-modality training. Either fix/reproduce OneLLM fairly or de-emphasize it and center Unimodal/Naive/X-InstructBLIP with matched compute and data.
  3. [§5.2 Tables 1–2; §5.3 ablations] No uncertainty quantification for the main results. Tables 1–2 and all ablations report single-run point estimates with no seeds, error bars, or significance tests, despite free parameters (λ, Ks/Kg, LoRA, LRs) and a claim of consistent outperformance across eight tasks and four modalities. For a methods paper whose contribution is empirical superiority under a new protocol, multi-seed means/std or at least two additional seeds on the primary Ego-in-Exo averages are needed to support “consistently outperforms.”
  4. [§4.2–4.3, Fig. 6, Conclusion] The mechanism for cross-modal transfer is under-validated relative to the claim. §4.2–4.3 state that modality-general probes interact with one modality per sample and transfer only via shared parameters under interleaved batches; joint multi-modal samples are avoided for context length. Fig. 6 shows L_dis increases inter/intra-probe variance and subspace distances, and Table 3 shows adding general probes helps under multi-training, but there is no direct test that general-probe updates from modality A improve modality B (e.g., train with ego+exo then ablate general-probe gradients from the auxiliary, or freeze general probes after multi-training and retrain specific probes unimodally). Given the Conclusion’s own limitation that MoP assumes a shared frozen encoder, the transfer story needs a causal or leave-one-auxiliary-out experiment, not only geometry diagnostics.
minor comments (7)
  1. [Figure 3] Figure 3 caption: “Ablation on prove disentanglement loss weight” → “probe”.
  2. [§5.2 Music-AVQA] §5.2 Music-AVQA paragraph cites “audio-only MLLM achieves 35.01 … naive … 39.28,” which does not match Table 2 averages (46.75 / 53.44). Align text with table.
  3. [Abstract, §5.2] Clarify how “up to 65% relative improvement” is computed (which baseline, which cell); absolute point gains in Tables 1–2 are clearer and should lead.
  4. [§4.3 Eq. (10)] Eq. (10): L_dis averages ∥D ⊙ S^m_l∥_1 over layers; state whether this is computed per-sample then batch-averaged, and whether general and specific blocks are both pushed apart (intended) or whether same-family probes should be allowed to correlate.
  5. [§5.1 Benchmarks] Depth is derived from exocentric video via DepthAnythingV2 (§5.1); discuss whether depth is an independent privileged stream or a processed view of exo, which affects interpretation of “four modalities” and depth-inference gains.
  6. [Appendix E, §5.1] Appendix Table 4 latency/VRAM is useful; note training cost (4×H200, 1 epoch) and whether probe cross-attention is applied at train only or also at inference for every layer.
  7. [§2] Related work on LUPI / cross-modal distillation is present but brief; a short comparison table of privileged vs missing-modality MLLM objectives would help position MoP against MissRAG and classical distillation.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: empirical architecture paper whose performance claims rest on held-out single-modality accuracy, not tautological derivations or fitted quantities renamed as predictions.

full rationale

The paper proposes MoP (modality-specific + modality-general probes interacting via cross-attention with intermediate layers of a frozen shared encoder, Eqs. 6–7) plus MoP-X (probe disentanglement loss L_dis in Eq. 10 plus interleaved batching) and evaluates it under the privileged-modality protocol on Ego-in-Exo Perception and Music-AVQA. All load-bearing claims are comparative accuracy numbers (Tables 1–2, ablations in Table 3 and Figs. 3–5) measured on held-out questions with single-modality inference; none reduce by construction to a fitted parameter or to a self-defined identity. The disentanglement loss is an explicit regularizer (L = L_AR + λ L_dis) whose effect is diagnosed post-hoc via probe variance and subspace distances (Fig. 6), not presented as a first-principles prediction of the target metric. Self-citations ([34] Ego-in-Exo benchmark, [35] VisCoP probing) supply data and related technique context; they do not supply a uniqueness theorem or ansatz that forces the reported gains. The architecture is self-contained against external baselines (unimodal MLLM, Naive MLLM, X-InstructBLIP, OneLLM) trained on the same data. Experimental confounding (missing unimodal-MoP control) is a validity concern outside the circularity criteria. Consequently the derivation chain contains no self-definitional, fitted-as-prediction, or load-bearing self-citation reductions.

Assumptions & free parameters 4 free parameters · 5 assumptions · 3 invented entities

The central empirical claim rests on a frozen shared encoder, hand-chosen probe counts and loss weight, and the hypothesis that intermediate specific/general probes plus interleaved single-modality batches transfer privileged information. Those modeling choices and free hyperparameters are load-bearing; the paper does not derive them from first principles.

free parameters (4)
  • probe disentanglement loss weight λ
    Chosen by ablation; λ=100 used for main results (Figure 3). Directly scales the regularizer that is claimed to prevent probe collapse.
  • number of modality-specific and modality-general probes (Ks, Kg)
    Default 8+8 unless otherwise specified; controls capacity of the invented probe interface.
  • LoRA rank r and learning rates
    r=16; connector/LoRA LR 1e-5; tokenizer/probe cross-attn LR 2e-6. Training dynamics depend on these hand-set values.
  • probe initialization variance
    Probes initialized from N(0, 0.02); affects early probe geometry before disentanglement.
assumptions (5)
  • domain assumption A single frozen universal modality encoder (here SigLIP) maps heterogeneous modalities into a shared intermediate space suitable for probing.
    Stated as design premise in Sections 4.1–4.2 and listed as a limitation in the Conclusion; without it MoP does not apply.
  • domain assumption Cross-modal transfer depends primarily on modality-general decisive features rather than non-transferable modality-specific signals (Modality Focusing Hypothesis).
    Invoked in Introduction and Section 4.2 to justify splitting specific vs general probes (cite Xue et al.).
  • domain assumption Intermediate encoder layers retain useful modality-dependent signals not preserved in final-layer alignment alone.
    Motivation for layer-wise probing (Introduction limitation (i); Section 4.2).
  • ad hoc to paper Off-diagonal cosine similarity of probe tokens is a valid proxy for unwanted probe collapse and should be minimized.
    Probe disentanglement loss L_dis (Eqs. 8–10) is introduced without independent theoretical guarantee that this objective yields transferable features.
  • ad hoc to paper Modality-interleaved single-modality batches suffice for cross-modal interaction via shared parameters (no joint multi-modal samples required).
    MoP-X batching strategy in Section 4.3; intentional design choice that underpins transfer claims.
invented entities (3)
  • Mixture of Probes (modality-specific + modality-general probe tokens with layer-wise cross-attention)
    purpose: Extract and organize intermediate encoder signals into disentangled specific and general representations for the LLM.
    Core architectural invention; independent_evidence false because utility is only shown via this paper’s task metrics.
  • Probe disentanglement loss L_dis
    purpose: Prevent probe collapse and encourage separation of specific vs general probes.
    New training objective (Eq. 10); supported only by ablations and probe variance/subspace plots in this work.
  • MoP Cross-modal Training (MoP-X)
    purpose: Combine L_dis with modality-interleaved batching for privileged-modality transfer.
    Named training strategy package; not independently validated outside the reported experiments.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Mixture of Probes: Learning from Privileged Modalities in Multimodal LLMs Through Probing." pith.science (2026). https://pith.science/paper/UMFTMXB7

@misc{pith2026260708839,
  author       = {Pith},
  title        = {Pith review of: Mixture of Probes: Learning from Privileged Modalities in Multimodal LLMs Through Probing},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/UMFTMXB7}},
  note         = {Machine review of arXiv:2607.08839}
}
read the original abstract

Multimodal Large Language Models (MLLMs) are typically designed under the assumption that all modalities available during training will also be accessible at inference. However, many real-world settings violate this assumption, requiring models to operate under a privileged modality setting, where auxiliary modalities are available only during training. While these modalities contain valuable information, existing MLLMs largely fail to leverage them effectively, as they treat modalities as interchangeable inputs rather than sources of complementary supervision. We propose Mixture of Probes (MoP), a novel framework that disentangles modality-specific and modality-general signals within the MLLM, allowing the model to preserve modality-dependent structure while learning transferable representations across modalities. At its core, MoP achieves this through a structured probing mechanism that extracts and organizes information from intermediate representations of a shared modality encoder, rather than relying only on final-layer alignment as done in existing MLLMs. To support this disentanglement, we further introduce MoP Cross-modal Training (MoP-X), a training strategy for MoP centered around a probe disentanglement loss that prevents probe collapse and encourages cross-modal learning. We evaluate MoP across two domains spanning eight tasks and four modalities under a comprehensive evaluation protocol tailored to the privileged modality setting, where each modality is independently treated as the sole input at inference time. MoP consistently outperforms strong MLLM baselines, achieving up to 65% relative improvement, demonstrating that auxiliary modalities, even when unavailable at inference, can provide substantial gains when effectively leveraged during training. Code, model checkpoints, and evaluation protocols will be made available at https://github.com/Sony/MoP.

Figures

Figures reproduced from arXiv: 2607.08839 by the authors.

Figure 1
Figure 1. (left) MoP improves single-modality inference across five modalities, outperforming both unimodal MLLMs and naive multimodal training. (right) MoP achieves this through a structured probing mechanism with modality-specific and modality-general probes, which disentangle modality representations before integration with the LLM. Abstract Multimodal Large Language Models (MLLMs) are typically designed under the assumpti… view at source ↗
Figure 2
Figure 2. MoP Architecture. In this example, three modalities are shown: egocentric, exocentric, and depth. Each sample in a training batch consists of a single modality, in this example that modality is the exocentric video. preserved in the final representation. Motivated by the modality focusing hypothesis [43], MoP learns two complementary probe families: modality-specific probes, which capture signals unique to each moda… view at source ↗
Figure 5
Figure 5. Ablation on applica￾tion of disentanglement loss. 5.3 Ablation Study We perform all ablations on the three inference settings of the Ego-in-Exo Perception benchmark [PITH_FULL_IMAGE:figures/full_fig_p008_5.png] view at source ↗
Figures from the paper (4 more)
Figure 6
Figure 6. Figure 6: Effect of probe disentanglement loss on learned probe representations. at every layer yields the strongest overall results, indicating that encouraging probe specialization throughout the full encoder is more effective than constraining it only locally. 5.4 Model Diagn…
Figure 7
Figure 7. Figure 7: Data scaling property of MoP. MoP data scaling [PITH_FULL_IMAGE:figures/full_fig_p009_7.png]
Figure 8
Figure 8. Figure 8: t-SNE visualization of specific and gen￾eral probes [PITH_FULL_IMAGE:figures/full_fig_p015_8.png]
Figure 9
Figure 9. Figure 9: Qualitative results of MoP on single modality inference. D Qualitative Results [PITH_FULL_IMAGE:figures/full_fig_p016_9.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

51 extracted references · 10 linked inside Pith

  1. [47]

    Videollama 3: Frontier multimodal foundation models for image and video understanding.arXiv preprint arXiv:2501.13106, 2025

    Boqiang Zhang, Kehan Li, Zesen Cheng, Zhiqiang Hu, Yuqian Yuan, Guanzheng Chen, Sicong Leng, Yuming Jiang, Hang Zhang, Xin Li, Peng Jin, Wenqi Zhang, Fan Wang, Lidong Bing, and Deli Zhao. Videollama 3: Frontier multimodal foundation models for image and video understanding.arXiv preprint arXiv:2501.13106, 2025. 7

    arXiv 2025

  2. [1]

    Soundnet: Learning sound representa- tions from unlabeled video

    Yusuf Aytar, Carl V ondrick, and Antonio Torralba. Soundnet: Learning sound representa- tions from unlabeled video. InProceedings of the 30th International Conference on Neural Information Processing Systems, NIPS’16, page 892–900, Red Hook, NY , USA, 2016. Curran Associates Inc. 3

    2016

  3. [2]

    Qwen2.5-vl technical report

    Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, Humen Zhong, Yuanzhi Zhu, Mingkun Yang, Zhaohai Li, Jianqiang Wan, Pengfei Wang, Wei Ding, Zheren Fu, Yiheng Xu, Jiabo Ye, Xi Zhang, Tianbao Xie, Zesen Cheng, Hang Zhang, Zhibo Yang, Haiyang Xu, and Junyang Lin. Qwen2.5-vl technical report. a...

    arXiv 2025

  4. [3]

    Paligemma: A versatile 3b vlm for transfer, 2024

    Lucas Beyer, Andreas Steiner, André Susano Pinto, Alexander Kolesnikov, Xiao Wang, Daniel Salz, Maxim Neumann, Ibrahim Alabdulmohsin, Michael Tschannen, Emanuele Bugliarello, Thomas Unterthiner, Daniel Keysers, Skanda Koppula, Fangyu Liu, Adam Grycner, Alexey Gritsenko, Neil Houlsby, Manoj Kumar, Keran Rong, Julian Eisenschlos, Rishabh Kabra, Matthias Bau...

    2024

  5. [4]

    Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-V oss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwi...

    2020

  6. [5]

    X-llm: Bootstrapping advanced large language models by treating multi-modalities as foreign languages.arXiv preprint arXiv:2305.04160, 2023

    Feilong Chen, Minglun Han, Haozhi Zhao, Qingyang Zhang, Jing Shi, Shuang Xu, and Bo Xu. X-llm: Bootstrapping advanced large language models by treating multi-modalities as foreign languages.arXiv preprint arXiv:2305.04160, 2023. 3

    arXiv 2023

  7. [6]

    Toyota smarthome untrimmed: Real-world untrimmed videos for activity detection.IEEE Transactions on Pattern Analysis and Machine Intelligence, 2022

    Rui Dai, Srijan Das, Saurav Sharma, Luca Minciullo, Lorenzo Garattoni, Francois Bremond, and Gianpiero Francesca. Toyota smarthome untrimmed: Real-world untrimmed videos for activity detection.IEEE Transactions on Pattern Analysis and Machine Intelligence, 2022. 2

    2022

  8. [7]

    Instructblip: Towards general-purpose vision-language models with instruction tuning

    Wenliang Dai, Junnan Li, Dongxu Li, Anthony Meng Huat Tiong, Junqi Zhao, Weisheng Wang, Boyang Li, Pascale Fung, and Steven Hoi. Instructblip: Towards general-purpose vision-language models with instruction tuning. InAdvances in Neural Information Processing Systems (NeurIPS), 2023. 3

    2023

Show all 51 references
  1. [8]

    Modality-specific and modality-general representations of subjective value in frontal cortex.Communications Biology, 7:1550, 2024

    Shilpa Dang, Jessica Emily Antono, Igor Kagan, and Arezoo Pooresmaeili. Modality-specific and modality-general representations of subjective value in frontal cortex.Communications Biology, 7:1550, 2024. 2

    2024

  2. [9]

    Toyota smarthome: Real-world activities of daily living

    Srijan Das, Rui Dai, Michal Koperski, Luca Minciullo, Lorenzo Garattoni, Francois Bremond, and Gianpiero Francesca. Toyota smarthome: Real-world activities of daily living. InProceed- ings of the IEEE/CVF International Conference on Computer Vision, pages 833–842, 2019. 2

    2019

  3. [10]

    Kristen Grauman, Andrew Westbury, Eugene Byrne, Zachary Chavis, Antonino Furnari, Ro- hit Girdhar, Jackson Hamburger, Hao Jiang, Miao Liu, Xingyu Liu, Miguel Martin, Tushar Nagarajan, Ilija Radosavovic, Santhosh Kumar Ramakrishnan, Fiona Ryan, Jayant Sharma, Michael Wray, Meng...

    2022

  4. [11]

    Kristen Grauman, Andrew Westbury, Lorenzo Torresani, Kris Kitani, Jitendra Malik, Tri- antafyllos Afouras, Kumar Ashutosh, Vijay Baiyya, Siddhant Bansal, Bikram Boote, Eugene Byrne, Zach Chavis, Joya Chen, Feng Cheng, Fu-Jen Chu, Sean Crane, Avijit Dasgupta, Jing Dong, Maria E...

    2025

  5. [12]

    Cross modal distillation for supervision transfer

    Saurabh Gupta, Judy Hoffman, and Jitendra Malik. Cross modal distillation for supervision transfer. InProceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016. 3

    2016

  6. [13]

    Onellm: One framework to align all modalities with language

    Jiaming Han, Kaixiong Gong, Yiyuan Zhang, Jiaqi Wang, Kaipeng Zhang, Dahua Lin, Yu Qiao, Peng Gao, and Xiangyu Yue. Onellm: One framework to align all modalities with language. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR),

  7. [14]

    Imagebind-llm: Multi-modality instruction tuning

    Jiaming Han, Renrui Zhang, Wenqi Shao, Peng Gao, Peng Xu, Han Xiao, Kaipeng Zhang, Chris Liu, Song Wen, Ziyu Guo, Xudong Lu, Shuai Ren, Yafei Wen, Xiaoxin Chen, Xiangyu Yue, Hongsheng Li, and Yu Qiao. Imagebind-llm: Multi-modality instruction tuning. InInternational Conference...

    2024

  8. [15]

    Enhanced computer vision with microsoft kinect sensor: A review.IEEE Transactions on Cybernetics, 43(5):1318–1334, 2013

    Jungong Han, Ling Shao, Dong Xu, and Jamie Shotton. Enhanced computer vision with microsoft kinect sensor: A review.IEEE Transactions on Cybernetics, 43(5):1318–1334, 2013. 2

    2013

  9. [16]

    Visual prompt tuning

    Menglin Jia, Luming Tang, Bor-Chun Chen, Claire Cardie, Serge Belongie, Bharath Hariharan, and Ser-Nam Lim. Visual prompt tuning. InECCV, 2022. 3

    2022

  10. [17]

    Learning to answer questions in dynamic audio-visual scenarios

    Guangyao li, Yake Wei, Yapeng Tian, Chenliang Xu, Ji-Rong Wen, and Di Hu. Learning to answer questions in dynamic audio-visual scenarios. InIEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2022. 7

    2022

  11. [18]

    Junnan Li, Dongxu Li, Silvio Savarese, and Steven C. H. Hoi. Blip-2: Bootstrapping language- image pre-training with frozen image encoders and large language models. InInternational Conference on Machine Learning, 2023. 3 11

    2023

  12. [19]

    Video-llava: Learning united visual representation by alignment before projection

    Bin Lin, Bin Zhu, Yang Ye, Munan Ning, Peng Jin, and Li Yuan. Video-llava: Learning united visual representation by alignment before projection. InConference on Empirical Methods in Natural Language Processing (EMNLP), 2024. 2

    2024

  13. [20]

    Visual instruction tuning

    Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. In Advances in Neural Information Processing Systems (NeurIPS), 2023. 3

    2023

  14. [21]

    Unifying distillation and privileged information

    David Lopez-Paz, Léon Bottou, Bernhard Schölkopf, and Vladimir Vapnik. Unifying distillation and privileged information. InInternational Conference on Learning Representations (ICLR),

  15. [22]

    Pretrained transformers as universal computation engines

    Kevin Lu, Aditya Grover, Pieter Abbeel, and Igor Mordatch. Pretrained transformers as universal computation engines. InProceedings of the AAAI Conference on Artificial Intelligence (AAAI),

  16. [23]

    Smil: Multimodal learning with severely missing modality

    Mengmeng Ma, Jian Ren, Long Zhao, Sergey Tulyakov, Cathy Wu, and Xi Peng. Smil: Multimodal learning with severely missing modality. InProceedings of the AAAI Conference on Artificial Intelligence (AAAI), 2021. 3

    2021

  17. [24]

    Video-chatgpt: Towards detailed video understanding via large vision and language models

    Muhammad Maaz, Hanoona Rasheed, Salman Khan, and Fahad Shahbaz Khan. Video-chatgpt: Towards detailed video understanding via large vision and language models. InProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (ACL), 2024. 3

    2024

  18. [25]

    Anymal: An efficient and scalable any-modality augmented language model

    Seungwhan Moon, Andrea Madotto, Zhaojiang Lin, Tushar Nagarajan, Matt Smith, Shashank Jain, Chun-Fu Yeh, Prakash Murugesan, Peyman Heidari, Yue Liu, Kavya Srinet, Babak Damavandi, and Anuj Kumar. Anymal: An efficient and scalable any-modality augmented language model. InProcee...

    2024

  19. [26]

    Taylor, and Florian Nebout

    Natalia Neverova, Christian Wolf, Graham W. Taylor, and Florian Nebout. Moddrop: Adap- tive multi-modal gesture recognition.IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI), 2016. 3

    2016

  20. [27]

    Gpt-4o system card, 2024

    OpenAI. Gpt-4o system card, 2024. 2

    2024

  21. [28]

    McDermott, William T

    Andrew Owens, Jiajun Wu, Josh H. McDermott, William T. Freeman, and Antonio Torralba. Ambient sound provides supervision for visual learning. InProceedings of the European Conference on Computer Vision (ECCV), 2016. 3

    2016

  22. [29]

    X-instructblip: A framework for aligning x-modal instruction-aware representations to llms and emergent cross-modal reasoning

    Artemis Panagopoulou, Le Xue, Ning Yu, Junnan Li, Dongxu Li, Shafiq Joty, Ran Xu, Silvio Savarese, Caiming Xiong, and Juan Carlos Niebles. X-instructblip: A framework for aligning x-modal instruction-aware representations to llms and emergent cross-modal reasoning. In Proceedi...

    2024

  23. [30]

    Missrag: Addressing the missing modality challenge in multimodal large language models

    Vittorio Pipoli, Alessia Saporita, Federico Bolelli, Marcella Cornia, Lorenzo Baraldi, Costantino Grana, Rita Cucchiara, and Elisa Ficarra. Missrag: Addressing the missing modality challenge in multimodal large language models. InProceedings of the IEEE/CVF International Confe...

    2025

  24. [31]

    Qwen2.5 technical report.arXiv preprint arXiv:2412.15115, 2025

    Qwen Team, An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jingren Zhou, Junyang Lin, Kai Dang, Keming Lu, Keqin Bao, Kexin Yang...

    2025 arXiv

  25. [32]

    Do vision transformers see like convolutional neural networks? InAdvances in Neural Information Processing Systems (NeurIPS), 2021

    Maithra Raghu, Thomas Unterthiner, Simon Kornblith, Chiyuan Zhang, and Alexey Dosovit- skiy. Do vision transformers see like convolutional neural networks? InAdvances in Neural Information Processing Systems (NeurIPS), 2021. 2 12

    2021

  26. [33]

    Llavidal: A large language-vision model for daily activities of living

    Dominick Reilly, Rajatsubhra Chakraborty, Arkaprava Sinha, Manish Kumar Govind, Pu Wang, Francois Bremond, Le Xue, and Srijan Das. Llavidal: A large language-vision model for daily activities of living. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Re...

    2025

  27. [34]

    From my view to yours: Ego-to-exo transfer in vlms for understanding activities of daily living.arXiv preprint arXiv:2501.05711, 2025

    Dominick Reilly, Manish Kumar Govind, Le Xue, and Srijan Das. From my view to yours: Ego-to-exo transfer in vlms for understanding activities of daily living.arXiv preprint arXiv:2501.05711, 2025. 6, 7

    2025 arXiv

  28. [35]

    Viscop: Visual probing for video domain adaptation of vision language models

    Dominick Reilly, Manish Kumar Govind, Le Xue, and Srijan Das. Viscop: Visual probing for video domain adaptation of vision language models. InProceedings of the European Conference on Computer Vision (ECCV), 2026. 2

    2026

  29. [36]

    Timechat: A time-sensitive multi- modal large language model for long video understanding.arXiv preprint arXiv:2312.02051,

    Shuhuai Ren, Linli Yao, Shicheng Li, Xu Sun, and Lu Hou. Timechat: A time-sensitive multi- modal large language model for long video understanding.arXiv preprint arXiv:2312.02051,

    arXiv

  30. [37]

    Pandagpt: One model to instruction-follow them all

    Yixuan Su, Tian Lan, Huayang Li, Jialu Xu, Yan Wang, and Deng Cai. Pandagpt: One model to instruction-follow them all. InProceedings of the 61st Annual Meeting of the Association for Computational Linguistics (ACL), 2023. 3

    2023

  31. [38]

    Gemma Team, Thomas Mesnard, Cassidy Hardin, Robert Dadashi, Surya Bhupatiraju, Shreya Pathak, Laurent Sifre, Morgane Rivière, Mihir Sanjay Kale, Juliette Love, Pouya Tafti, Léonard Hussenot, Pier Giuseppe Sessa, Aakanksha Chowdhery, Adam Roberts, Aditya Barua, Alex Botev, Alex...

    2024

  32. [39]

    Llama: Open and efficient foundation language models.arXiv preprint arXiv:2302.13971, 2023

    Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timo- thée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. Llama: Open and efficient foundation langua...

    2023 arXiv

  33. [40]

    Learning using privileged information: Similarity control and knowledge transfer

    Vladimir Vapnik and Rauf Izmailov. Learning using privileged information: Similarity control and knowledge transfer. InProceedings of the 32nd International Conference on Machine Learning (ICML), 2015. 3

    2015

  34. [41]

    Deep multimodal learning with missing modality: A survey.Transactions on Machine Learning Research (TMLR), 2026

    Renjie Wu, Hu Wang, Hsiang-Ting Chen, and Gustavo Carneiro. Deep multimodal learning with missing modality: A survey.Transactions on Machine Learning Research (TMLR), 2026. 3

    2026

  35. [42]

    Qwen2.5-omni technical report.arXiv preprint arXiv:2503.20215, 2025

    Jin Xu, Zhifang Guo, Jinzheng He, Hangrui Hu, Ting He, Shuai Bai, Keqin Chen, Jialin Wang, Yang Fan, Kai Dang, Bin Zhang, Xiong Wang, Yunfei Chu, and Junyang Lin. Qwen2.5-omni technical report.arXiv preprint arXiv:2503.20215, 2025. 2

    2025 arXiv

  36. [43]

    The modality focusing hypothesis: Towards understanding crossmodal knowledge distillation

    Zihui Xue, Zhengqi Gao, Sucheng Ren, and Hang Zhao. The modality focusing hypothesis: Towards understanding crossmodal knowledge distillation. InInternational Conference on Learning Representations (ICLR), 2023. 2, 4, 5 13

    2023

  37. [44]

    Depth anything v2

    Lihe Yang, Bingyi Kang, Zilong Huang, Zhen Zhao, Xiaogang Xu, Jiashi Feng, and Hengshuang Zhao. Depth anything v2. InAdvances in Neural Information Processing Systems, 2024. 6

    2024

  38. [45]

    mplug-owl2: Revolutionizing multi-modal large language model with modality collaboration

    Qinghao Ye, Haiyang Xu, Jiabo Ye, Ming Yan, Anwen Hu, Haowei Liu, Qi Qian, Ji Zhang, Fei Huang, and Jingren Zhou. mplug-owl2: Revolutionizing multi-modal large language model with modality collaboration. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern R...

    2024

  39. [46]

    Sigmoid loss for language image pre-training

    Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, and Lucas Beyer. Sigmoid loss for language image pre-training. InProceedings of the IEEE/CVF International Conference on Computer Vision. IEEE, 2023. 7

    2023

  40. [48]

    Video-llama: An instruction-tuned audio-visual language model for video understanding

    Hang Zhang, Xin Li, and Lidong Bing. Video-llama: An instruction-tuned audio-visual language model for video understanding. InProceedings of the 2023 Conference on Empirical Methods in Natural Language Processing (EMNLP), 2023. 2, 3

    2023

  41. [49]

    Video-llama: An instruction-tuned audio-visual language model for video understanding

    Hang Zhang, Xin Li, and Lidong Bing. Video-llama: An instruction-tuned audio-visual language model for video understanding. InConference on Empirical Methods in Natural Language Processing (EMNLP), 2023. 2

    2023

  42. [50]

    Meta-transformer: A unified framework for multimodal learning.arXiv preprint arXiv:2307.10802, 2023

    Yiyuan Zhang, Kaixiong Gong, Kaipeng Zhang, Hongsheng Li, Yu Qiao, Wanli Ouyang, and Xiangyu Yue. Meta-transformer: A unified framework for multimodal learning.arXiv preprint arXiv:2307.10802, 2023. 3

    2023 arXiv

  43. [51]

    Chatbridge: Bridging modalities with large language model as a language catalyst

    Zijia Zhao, Longteng Guo, Tongtian Yue, Sihan Chen, Shuai Shao, Xinxin Zhu, Zehuan Yuan, and Jing Liu. Chatbridge: Bridging modalities with large language model as a language catalyst. arXiv preprint arXiv:2305.16103, 2023. 2, 3, 6 14 Appendix A Broader impacts The broader imp...

    2023 arXiv

Pith tools

Reviewed July 13, 2026 · model on record in the stance chip above.