Pith. sign in

REVIEW 3 major objections 5 minor 1 cited by

BASIC: Boosting Visual Alignment with Intrinsic Refined Embeddings in Multimodal Large Language Models

T0 review · 3 major / 5 minor · reviewed 2026-08-05 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read BASIC improves multimodal LLMs by directly supervising initial visual embeddings with the LLM's own refined internal embeddings, adding no extra models or annotations.

desk verdict A well-ablated self-distillation recipe for MLLM visual alignment that likely works, but the central mechanism is less proven than the benchmark gains suggest. read the letter →

arxiv 2508.06895 v1 pith:QAJKZ4KF submitted 2025-08-09 cs.CV cs.AI

Jianting Tang , Yubo Wang , Haoyu Cao , Linli Xu This is my paper
classification cs.CVcs.AI
keywords multimodallargelanguagemodelsvision-languagealignmentdirectvisualsupervisionrefinedembeddingsself-distillationdirectionalsemanticdistributionLLaVA
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Standard multimodal LLM training supervises only text outputs, leaving the vision projector's embeddings without direct feedback. This paper argues that the LLM's own shallow layers refine those visual embeddings into semantically meaningful tokens, and that these refined embeddings make a better, free supervision target for the projector. The proposed BASIC method adds two losses to the usual autoregressive loss: one that narrows the angle between initial and refined embeddings on the unit sphere, and one that matches their vocabulary logit distributions through KL divergence. With the same data and architecture as LLaVA-1.5, BASIC consistently improves eight vision-language benchmarks, suggesting that direct visual supervision is a neglected source of alignment.

What carries the argument

The central mechanism is the intrinsic refined embedding $\tilde{V}_i$: a weighted sum (with quadratically increasing weights) of the hidden states of image tokens from the LLM's shallow layers, which acts as a self-generated supervision target. Two objectives carry the alignment: $\mathcal{L}_{\mathrm{dir}} = \sum_i a_i \| \hat{v}_i - \hat{\tilde{v}}_i \|^2$ on the unit hypersphere, aligning directions, and $\mathcal{L}_{\mathrm{sem}} = \sum_i a_i \operatorname{KL}(P_i \| Q_i)$, where $P_i$ and $Q_i$ are the vocabulary softmax distributions of the supervisory and initial embeddings. Each patch's weight $a_i$ is the mean text-to-image attention across layers, so patches that matter more for

What would settle it

A decisive control: train BASIC with the same two losses but replace the supervisory embedding with (i) a random token embedding or (ii) the initial embedding plus isotropic noise; if gains persist under (i) or vanish under (ii), the specific claim that the LLM's refined shallow-layer embeddings are the effective target is falsified. A simpler check: record the mean cosine similarity between initial and supervisory embeddings before and after training—it should rise substantially if the directional objective is doing its named work.

Watch

Extended reading notes

Core claim

The central claim is that the LLM's shallow-layer refinement of visual embeddings provides a free, better supervision target for the vision projector. Observing that initial visual embeddings often match irregular text tokens while shallow-layer embeddings match meaningful ones (color, shape, object class), the authors define the supervisory embedding $\tilde{V}_i$ as a quadratically weighted sum of those refined embeddings. Two objectives guide the projector: $\mathcal{L}_{\mathrm{dir}}$, the $\ell^2$ distance between normalized initial and supervisory embeddings, and $\mathcal{L}_{\mathrm{sem}}$, the KL divergence between their vocabulary logit distributions. The total loss $\mathcal{L} =

Load-bearing premise

The method assumes that the refined visual embeddings in the LLM's shallow layers are a genuinely better semantic target, and not merely smoothed or averaged toward contextually frequent tokens; if that assumption fails, the added losses could pull the projector toward a less informative target.

Editorial extensions

If this is right

  • If the claim is right, standard MLLM training is leaving a free signal on the table: the LLM's own shallow layers already compute better-aligned visual embeddings, and supervising the projector toward them improves downstream reasoning.
  • Both the directional and the semantic-distribution losses are individually useful, and combined they outperform either alone (Table 2), so the improvement is not an artifact of one objective.
  • The gains reproduce across different vision encoders (CLIP-L, SigLIP-SO) and LLMs from 2B to 13B (Table 3), suggesting the mechanism generalizes across MLLM architectures.
  • The approach adds no annotation or inference cost beyond the training-time loss; it only requires access to the LLM's hidden states.
  • A known cost is a small drop on TextVQA, a reading-heavy benchmark, which the authors attribute to semantic supervision blurring tiny in-image text.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • One testable extension: because the supervision target is itself the model's output, BASIC may act as a regularizer that reduces the modality gap; one could measure whether it also improves robustness to distribution shift or reduces hallucination, not just benchmark average.
  • If shallow-layer refinement reflects contextual smoothing, the same recipe could be applied to other continuous modalities (audio, video) where a frozen backbone produces refined representations before task heads.
  • The attention-weighting of supervision (Eq 5) suggests a natural curriculum: patches that the text most attends to are trained hardest; a stronger hypothesis is that this weighting, not the loss itself, causes most of the gain—testable by keeping equal weights and reweighting randomly.
  • A potential risk the authors do not explore: the refined target may pull initial embeddings toward contextually generic tokens; if so, the gains may shrink on fine-grained or rare-object tasks, and the method might benefit from an entropy or diversity term.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper proposes BASIC, a training method for multimodal large language models that adds direct supervision to the vision projector output. The authors observe that visual embeddings become more semantically aligned with text tokens in the LLM's shallow layers, and use a weighted sum of those refined embeddings, Eq. (4), as a supervision target. Two auxiliary losses are added to the autoregressive objective, Eq. (8): a normalized-L2/angular loss, Eq. (6), and a KL divergence between full-vocabulary logit distributions, Eq. (7), weighted per patch by text-to-image attention, Eq. (5). Experiments on LLaVA-1.5 with Vicuna-7B/13B and with other vision-encoder/LLM combinations show consistent improvements on most of eight benchmarks, with a small decline on TextVQA. Ablations cover each loss, the layer range, the layer-weighting scheme, and the attention-based patch weighting.

Significance. If the reported gains are reproducible, BASIC is a simple, broadly applicable training modification that requires no additional labels or models; the ablation coverage is more complete than typical for this area, and the gains are consistent across several architecture combinations. The main weakness is that the interpretability evidence for 'semantic refinement' is qualitative and potentially biased by token frequency, so the auxiliary losses may partly act as generic regularizers rather than as faithful semantic super vision. Nonetheless, the benchmark contribution is credible, and the paper includes useful falsifiable comparisons such as the layer-source ablation.

major comments (3)
  1. [§3.2, Eq. (2), §5.2] The claim that shallow LLM layers 'refine' visual embeddings is supported only by nearest-token cosine matches and a manual count on 30 images by 2 students (74/576 vs. 217/576 meaningful tokens). Nearest-token matching is known to be biased by token frequency and embedding anisotropy; the paper itself notes that deep layers collapse to '</s>'. Since Eqs. (6)-(7) are pure matching losses, they could improve benchmarks by acting as a generic smoothness/self-distillation regularizer rather than by encoding semantically more informative targets. Because the target construction is the paper's core motivation, please add a control that isolates target content: e.g., apply the same losses to random or input-independent targets, to smoothed initial embeddings, or to frequency-matched random tokens. Figure 6 (deep-layer targets degrade) is a useful partial control, but it does not rule out frequ
  2. [Eq. (4)-(8), Supp. B] Supplementary Section B states that L_dir and L_sem 'only influence the gradients of the vision projector parameters,' but no stop-gradient/detach operator appears in Eqs. (4), (6), or (7). In Stage 2 the LLM is trainable; without detaching the refined embeddings, the auxiliary losses will also update the LLM layers that produce the target, changing the target during training and contradicting the stated design. Please specify explicitly where gradients are stopped and confirm that the implementation detaches V~ before computing Eqs. (6)-(7). This is essential for reproducibility and for interpreting the self-distillation claim.
  3. [Table 1, §5.2] BASIC declines on TextVQA relative to the LLaVA baseline in several configurations (e.g., the 7B row drops from 58.2 to 58.0). The explanation in Sec. 5.2—that semantic-concept supervision blurs small text—is post hoc and untested. Given the abstract's claim of improvement 'across a wide range of benchmarks,' the paper should either provide a quantitative test of this explanation or temper the claim and discuss the trade-off explicitly. This is a caveat rather than a refutation, but it needs to be handled more carefully.
minor comments (5)
  1. [Throughout] No error bars or repeated-seed results are reported; the field often accepts single runs, but the number of runs and any variance should at least be stated.
  2. [§5.3, Table 4] Hyperparameters λ1=1 and λ2=0.01 are fixed without sensitivity analysis; please report how they were chosen and whether results are stable to modest perturbations.
  3. [§5.2] The manual evaluation of 'meaningful' embeddings lacks inter-annotator agreement and a detailed sampling protocol; if retained, add agreement metrics and clarify how images were selected and how 'meaningful' was defined.
  4. [§4.1, Eq. (5)] The notation in Eq. (5) is hard to follow: the index ranges, the layer set over which attention scores are averaged, and the definition of a_i^h should be stated precisely.
  5. [§2.2, Eqs. (4)-(7)] Typos and OCR artifacts should be cleaned: 'interpretility' in Sec. 2.2, missing math symbols in Eqs. (4)-(7), and inconsistent benchmark abbreviations (e.g., 'VQA T').

Circularity Check

0 steps flagged · score 1.0 of 10

No circular derivation: the headline benchmark gains are external, and the self-distillation construction is explicitly acknowledged.

full rationale

The paper's central claim is empirical: adding L_dir and L_sem (Eqs. 6-8) to LLaVA-1.5 training improves scores on standard, external benchmarks (Tables 1 and 3). The supervisory target V~ (Eq. 4) is indeed derived from the same model's shallow-layer activations of the same input embeddings, so L_dir and L_sem are self-consistency or self-distillation objectives rather than externally grounded labels. However, the paper explicitly frames this as self-distillation (Sec. 2.3: 'can be regarded as a form of self-distillation') and does not claim the target is an independent ground truth. No parameter is fitted to the evaluation benchmarks and then relabeled as a prediction; all reported benchmark results are comparisons against external test sets. The nearest-token visualizations and the 30-image manual count (Sec. 3.2, 5.2) are presented as interpretability and explanatory evidence, not as the derivation of the headline result. Even though that evidence is not frequency-controlled and is partly aligned with the training objective, it is not the load-bearing proof of the paper's main claim. There are no author self-citations, no imported uniqueness theorems, and no ansatz smuggled in via self-citation. Thus the derivation chain does not reduce to its own inputs by construction.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

The method introduces no new model components or artificial annotations. Its main burden is the set of internal-consistency assumptions listed above, especially that shallow-layer refined embeddings are reliable teachers. Free parameters are confined to loss weights and layer/weighting choices, all selected by ablations.

free parameters (4)
  • lambda1 (directional loss weight) = 1
    Chosen manually in Eq (8); no sensitivity ablation is reported.
  • lambda2 (semantic KL loss weight) = 0.01
    Chosen manually in Eq (8); no sensitivity ablation is reported.
  • Supervision layer range k = lower half: about layers 16-32 for 7B and 20-40 for 13B
    Selected by ablation in Fig 6; using deeper layers degrades performance.
  • Layer weight exponent = 2 (quadratically increasing)
    Table 4 ablation shows increasing quadratic weights beat constant or decreasing weights.
assumptions (4)
  • domain assumption Refined visual embeddings in the LLM's shallow layers are semantically better aligned with text than the initial projector embeddings.
    Core premise for using them as supervision; supported by qualitative nearest-token visualizations (Sec 3.2, Fig 2) and a 30-image manual count, not by an independent quantitative measure.
  • domain assumption Inner products of a visual embedding with all vocabulary embeddings form an interpretable semantic distribution, and KL divergence between such distributions is a valid alignment objective.
    Sec 4.3 assumes P=softmax(V~E^T) and Q=softmax(VE^T) capture semantic matching; no evidence that these logits are calibrated.
  • ad hoc to paper Text-to-image attention scores measure the importance of each image patch for supervision.
    Eq (5) uses mean attention from text tokens to image patches as a patch importance weight; this heuristic is not validated against alternative importance measures.
  • standard math Cosine similarity on the unit hypersphere captures semantic direction.
    Standard geometric assumption cited from interpretability literature.

how reviews work

0 comments
Cite this review

Pith. "Pith review of BASIC: Boosting Visual Alignment with Intrinsic Refined Embeddings in Multimodal Large Language Models." pith.science (2026). https://pith.science/paper/QAJKZ4KF

@misc{pith2026250806895,
  author       = {Pith},
  title        = {Pith review of: BASIC: Boosting Visual Alignment with Intrinsic Refined Embeddings in Multimodal Large Language Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/QAJKZ4KF}},
  note         = {Machine review of arXiv:2508.06895}
}
read the original abstract

Mainstream Multimodal Large Language Models (MLLMs) achieve visual understanding by using a vision projector to bridge well-pretrained vision encoders and large language models (LLMs). The inherent gap between visual and textual modalities makes the embeddings from the vision projector critical for visual comprehension. However, current alignment approaches treat visual embeddings as contextual cues and merely apply auto-regressive supervision to textual outputs, neglecting the necessity of introducing equivalent direct visual supervision, which hinders the potential finer alignment of visual embeddings. In this paper, based on our analysis of the refinement process of visual embeddings in the LLM's shallow layers, we propose BASIC, a method that utilizes refined visual embeddings within the LLM as supervision to directly guide the projector in generating initial visual embeddings. Specifically, the guidance is conducted from two perspectives: (i) optimizing embedding directions by reducing angles between initial and supervisory embeddings in semantic space; (ii) improving semantic matching by minimizing disparities between the logit distributions of both visual embeddings. Without additional supervisory models or artificial annotations, BASIC significantly improves the performance of MLLMs across a wide range of benchmarks, demonstrating the effectiveness of our introduced direct visual supervision.

Discussion (0). Sign in to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. RespiraMFM: A Multimodal Foundation Model with Contrastive Audio-Language Alignment for Respiratory Disease Identification

    cs.SD 2026-06 unverdicted novelty 5.0 of 10

    RespiraMFM reports 9.15% AUROC gain in supervised fine-tuning and 20.98% in zero-shot settings over baselines by aligning respiratory audio with clinical text across seven real-world datasets for five diseases.

Reference graph

Works this paper leans on

72 extracted references · 36 canonical work pages · cited by 1 Pith paper

  1. [1]

    Phi-3 technical report: A highly capable language model locally on your phone

    Marah Abdin, Jyoti Aneja, Hany Awadalla, Ahmed Awadal- lah, Ammar Ahmad Awan, Nguyen Bach, Amit Bahree, Arash Bakhtiari, Jianmin Bao, Harkirat Behl, et al. Phi-3 technical report: A highly capable language model locally on your phone. arXiv preprint arXiv:2404.14219, 2024. 8

    arXiv 2024

  2. [2]

    Flamingo: a visual language model for few-shot learning

    Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Men- sch, Katherine Millican, Malcolm Reynolds, et al. Flamingo: a visual language model for few-shot learning. Advances in neural information processing systems, 35:23716–23736,

  3. [3]

    Qwen-vl: A versatile vision-language model for un- derstanding, localization, text reading, and beyond

    Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. Qwen-vl: A versatile vision-language model for un- derstanding, localization, text reading, and beyond. arXiv preprint arXiv:2308.12966, 1(2):3, 2023. 6

    arXiv 2023

  4. [4]

    Beit: Bert pre-training of image transformers

    Hangbo Bao, Li Dong, Songhao Piao, and Furu Wei. Beit: Bert pre-training of image transformers. arXiv preprint arXiv:2106.08254, 2021. 2

    arXiv 2021

  5. [5]

    Introducing our multimodal models, 2023

    Rohan Bavishi, Erich Elsen, Curtis Hawthorne, Maxwell Nye, Augustus Odena, Arushi Somani, and Sa ˘gnak Tas ¸ırlar. Introducing our multimodal models, 2023. 6

    2023

  6. [6]

    Eliciting latent predictions from transformers with the tuned lens

    Nora Belrose, Zach Furman, Logan Smith, Danny Halawi, Igor Ostrovsky, Lev McKinney, Stella Biderman, and Ja- cob Steinhardt. Eliciting latent predictions from transformers with the tuned lens. arXiv preprint arXiv:2303.08112, 2023. 3

    arXiv 2023

  7. [7]

    Mechanistic interpretability for ai safety–a review

    Leonard Bereska and Efstratios Gavves. Mechanistic interpretability for ai safety–a review. arXiv preprint arXiv:2404.14082, 2024. 2

    arXiv 2024

  8. [8]

    Towards monose- manticity: Decomposing language models with dictionary learning

    Trenton Bricken, Adly Templeton, Joshua Batson, Brian Chen, Adam Jermyn, Tom Conerly, Nick Turner, Cem Anil, Carson Denison, Amanda Askell, et al. Towards monose- manticity: Decomposing language models with dictionary learning. Transformer Circuits Thread, 2, 2023. 3

    work page 2023

Show all 72 references
  1. [9]

    Internlm2 technical report

    Zheng Cai, Maosong Cao, Haojiong Chen, Kai Chen, Keyu Chen, Xin Chen, Xun Chen, Zehui Chen, Zhi Chen, Pei Chu, et al. Internlm2 technical report. arXiv preprint arXiv:2403.17297, 2024. 1

    2024 arXiv

  2. [10]

    Honeybee: Locality-enhanced projector for multimodal llm

    Junbum Cha, Wooyoung Kang, Jonghwan Mun, and Byungseok Roh. Honeybee: Locality-enhanced projector for multimodal llm. In Proceedings of the IEEE/CVF Con- ference on Computer Vision and Pattern Recognition, pages 13817–13827, 2024. 1

    2024

  3. [11]

    Sssd: Self-supervised self distillation

    Wei-Chi Chen and Wei-Ta Chu. Sssd: Self-supervised self distillation. In Proceedings of the IEEE/CVF Winter Confer- ence on Applications of Computer Vision, pages 2770–2777,

  4. [12]

    How far are we to gpt-4v? closing the gap to commercial multimodal models with open-source suites

    Zhe Chen, Weiyun Wang, Hao Tian, Shenglong Ye, Zhang- wei Gao, Erfei Cui, Wenwen Tong, Kongzhi Hu, Jiapeng Luo, Zheng Ma, et al. How far are we to gpt-4v? closing the gap to commercial multimodal models with open-source suites. arXiv preprint arXiv:2404.16821, 2024. 1, 2

    2024 arXiv

  5. [13]

    Gonzalez, Ion Stoica, and Eric P

    Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhang- hao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yong- hao Zhuang, Joseph E. Gonzalez, Ion Stoica, and Eric P. Xing. Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality, 2023. 1, 5, 7, 8

    2023

  6. [14]

    On the efficacy of knowledge distillation

    Jang Hyun Cho and Bharath Hariharan. On the efficacy of knowledge distillation. In Proceedings of the IEEE/CVF international conference on computer vision , pages 4794– 4802, 2019. 3

    2019

  7. [15]

    Towards automated circuit discovery for mechanistic interpretability

    Arthur Conmy, Augustine Mavor-Parker, Aengus Lynch, Stefan Heimersheim, and Adri `a Garriga-Alonso. Towards automated circuit discovery for mechanistic interpretability. Advances in Neural Information Processing Systems , 36: 16318–16352, 2023. 2

    2023

  8. [16]

    Sparse autoencoders find highly interpretable features in language models

    Hoagy Cunningham, Aidan Ewart, Logan Riggs, Robert Huben, and Lee Sharkey. Sparse autoencoders find highly interpretable features in language models. arXiv preprint arXiv:2309.08600, 2023. 3

    2023 arXiv

  9. [17]

    Wenliang Dai, Junnan Li, Dongxu Li, Anthony Meng Huat Tiong, Junqi Zhao, Weisheng Wang, Boyang Li, Pascale Fung, and Steven C. H. Hoi. Instructblip: Towards general- purpose vision-language models with instruction tuning. In NeurIPS, 2023. 6

    2023

  10. [18]

    The llama 3 herd of models

    Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Ab- hishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783 ,

    arXiv

  11. [19]

    Softmax linear units

    Nelson Elhage, Tristan Hume, Catherine Olsson, Neel Nanda, Tom Henighan, Scott Johnston, Sheer ElShowk, Nicholas Joseph, Nova DasSarma, Ben Mann, Danny Her- nandez, Amanda Askell, Kamal Ndousse, Andy Jones, et al. Softmax linear units. Transformer Circuits Thread, 2022. https:...

    2022

  12. [20]

    Making llama see and draw with seed tokenizer

    Yuying Ge, Sijie Zhao, Ziyun Zeng, Yixiao Ge, Chen Li, Xintao Wang, and Ying Shan. Making llama see and draw with seed tokenizer. arXiv preprint arXiv:2310.01218, 2023. 2

    2023 arXiv

  13. [21]

    Knowledge distillation: A survey

    Jianping Gou, Baosheng Yu, Stephen J Maybank, and Dacheng Tao. Knowledge distillation: A survey. Interna- tional Journal of Computer Vision, 129(6):1789–1819, 2021. 3

    2021

  14. [22]

    Making the v in vqa matter: Elevating the role of image understanding in visual question answer- ing

    Yash Goyal, Tejas Khot, Douglas Summers-Stay, Dhruv Ba- tra, and Devi Parikh. Making the v in vqa matter: Elevating the role of image understanding in visual question answer- ing. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 6904–6913...

    2017

  15. [23]

    Vizwiz grand challenge: Answering visual questions from blind people

    Danna Gurari, Qing Li, Abigale J Stangl, Anhong Guo, Chi Lin, Kristen Grauman, Jiebo Luo, and Jeffrey P Bigham. Vizwiz grand challenge: Answering visual questions from blind people. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3608–3617,

  16. [24]

    Learning lightweight lane detection cnns by self at- 9 tention distillation

    Yuenan Hou, Zheng Ma, Chunxiao Liu, and Chen Change Loy. Learning lightweight lane detection cnns by self at- 9 tention distillation. In Proceedings of the IEEE/CVF inter- national conference on computer vision , pages 1013–1021,

  17. [25]

    Gqa: A new dataset for real-world visual reasoning and compositional question answering

    Drew A Hudson and Christopher D Manning. Gqa: A new dataset for real-world visual reasoning and compositional question answering. In Proceedings of the IEEE/CVF con- ference on computer vision and pattern recognition , pages 6700–6709, 2019. 6

    2019

  18. [26]

    Perceiver: General perception with iterative attention

    Andrew Jaegle, Felix Gimeno, Andy Brock, Oriol Vinyals, Andrew Zisserman, and Joao Carreira. Perceiver: General perception with iterative attention. In International confer- ence on machine learning, pages 4651–4664. PMLR, 2021. 2

    2021

  19. [27]

    Mistral 7b

    Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lam- ple, Lucile Saulnier, et al. Mistral 7b. arXiv preprint arXiv:2310.06825, 2023. 8

    2023 arXiv

  20. [28]

    Unified language-vision pretraining in LLM with dynamic discrete visual tokenization

    Yang Jin, Kun Xu, Kun Xu, Liwei Chen, Chao Liao, Jian- chao Tan, Quzhe Huang, Bin Chen, Chengru Song, Dai Meng, Di Zhang, Wenwu Ou, Kun Gai, and Yadong Mu. Unified language-vision pretraining in LLM with dynamic discrete visual tokenization. In ICLR. OpenReview.net,

  21. [29]

    Referitgame: Referring to objects in pho- tographs of natural scenes

    Sahar Kazemzadeh, Vicente Ordonez, Mark Matten, and Tamara Berg. Referitgame: Referring to objects in pho- tographs of natural scenes. In Proceedings of the 2014 con- ference on empirical methods in natural language processing (EMNLP), pages 787–798, 2014. 6

    2014

  22. [30]

    Segment any- thing

    Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer White- head, Alexander C Berg, Wan-Yen Lo, et al. Segment any- thing. In Proceedings of the IEEE/CVF International Con- ference on Computer Vision, pages 4015–4026, 2023. 1

    2023

  23. [31]

    Obelics: An open web-scale filtered dataset of interleaved image-text documents

    Hugo Laurenc ¸on, Lucile Saulnier, L´eo Tronchon, Stas Bek- man, Amanpreet Singh, Anton Lozhkov, Thomas Wang, Sid- dharth Karamcheti, Alexander Rush, Douwe Kiela, et al. Obelics: An open web-scale filtered dataset of interleaved image-text documents. Advances in Neural Informa...

    2024

  24. [32]

    Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models

    Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. In In- ternational conference on machine learning , pages 19730– 19742. PMLR, 2023. 1, 2, 6

    2023

  25. [33]

    Mind the gap: Understanding the modality gap in multi-modal contrastive representation learning

    Victor Weixin Liang, Yuhui Zhang, Yongchan Kwon, Ser- ena Yeung, and James Y Zou. Mind the gap: Understanding the modality gap in multi-modal contrastive representation learning. Advances in Neural Information Processing Sys- tems, 35:17612–17625, 2022. 1

    2022

  26. [34]

    Does circuit analysis interpretability scale? evidence from mul- tiple choice capabilities in chinchilla

    Tom Lieberum, Matthew Rahtz, J ´anos Kram´ar, Neel Nanda, Geoffrey Irving, Rohin Shah, and Vladimir Mikulik. Does circuit analysis interpretability scale? evidence from mul- tiple choice capabilities in chinchilla. arXiv preprint arXiv:2307.09458, 2023. 3, 4

    2023 arXiv

  27. [35]

    Improved baselines with visual instruction tuning

    Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning. In Pro- ceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 26296–26306, 2024. 1, 2, 5, 6, 7, 8, 3, 4

    2024

  28. [36]

    Visual instruction tuning

    Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. Advances in neural information processing systems, 36, 2024. 1, 2

    2024

  29. [37]

    Mmbench: Is your multi-modal model an all-around player? In European Conference on Computer Vision, pages 216–233

    Yuan Liu, Haodong Duan, Yuanhan Zhang, Bo Li, Songyang Zhang, Wangbo Zhao, Yike Yuan, Jiaqi Wang, Conghui He, Ziwei Liu, et al. Mmbench: Is your multi-modal model an all-around player? In European Conference on Computer Vision, pages 216–233. Springer, 2025. 6

    2025

  30. [38]

    Learn to explain: Multimodal reasoning via thought chains for science question answering

    Pan Lu, Swaroop Mishra, Tanglin Xia, Liang Qiu, Kai-Wei Chang, Song-Chun Zhu, Oyvind Tafjord, Peter Clark, and Ashwin Kalyan. Learn to explain: Multimodal reasoning via thought chains for science question answering. Advances in Neural Information Processing Systems , 35:2507–2521,

  31. [39]

    Generation and comprehension of unambiguous object descriptions

    Junhua Mao, Jonathan Huang, Alexander Toshev, Oana Camburu, Alan L Yuille, and Kevin Murphy. Generation and comprehension of unambiguous object descriptions. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 11–20, 2016. 6

    2016

  32. [40]

    Self-distillation amplifies regularization in hilbert space

    Hossein Mobahi, Mehrdad Farajtabar, and Peter Bartlett. Self-distillation amplifies regularization in hilbert space. Ad- vances in Neural Information Processing Systems, 33:3351– 3361, 2020. 3

    2020

  33. [41]

    Dinov2: Learning robust visual features without supervision

    Maxime Oquab, Timoth ´ee Darcet, Th ´eo Moutakanni, Huy V o, Marc Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, et al. Dinov2: Learning robust visual features without supervision. arXiv preprint arXiv:2304.07193, 2023. 1

    2023 arXiv

  34. [42]

    Towards vision-language mechanistic interpretabil- ity: A causal tracing tool for blip

    Vedant Palit, Rohan Pandey, Aryaman Arora, and Paul Pu Liang. Towards vision-language mechanistic interpretabil- ity: A causal tracing tool for blip. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 2856–2861, 2023. 2

    2023

  35. [43]

    Bridg- ing vision and language spaces with assignment prediction

    Jungin Park, Jiyoung Lee, and Kwanghoon Sohn. Bridg- ing vision and language spaces with assignment prediction. arXiv preprint arXiv:2404.09632, 2024. 1

    2024 arXiv

  36. [44]

    Relational knowledge distillation

    Wonpyo Park, Dongju Kim, Yan Lu, and Minsu Cho. Relational knowledge distillation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 3967–3976, 2019. 3

    2019

  37. [45]

    Multi-modal auto-regressive modeling via visual words

    Tianshuo Peng, Zuchao Li, Lefei Zhang, Hai Zhao, Ping Wang, and Bo Du. Multi-modal auto-regressive modeling via visual words. CoRR, abs/2403.07720, 2024. 1, 6

    2024 arXiv

  38. [46]

    Beit v2: Masked image modeling with vector-quantized visual tokenizers

    Zhiliang Peng, Li Dong, Hangbo Bao, Qixiang Ye, and Furu Wei. Beit v2: Masked image modeling with vector-quantized visual tokenizers. arXiv preprint arXiv:2208.06366, 2022. 2

    2022 arXiv

  39. [47]

    Distillation-based training for multi-exit architectures

    Mary Phuong and Christoph H Lampert. Distillation-based training for multi-exit architectures. In Proceedings of the IEEE/CVF international conference on computer vision, pages 1355–1364, 2019. 3

    2019

  40. [48]

    Learning transferable visual models from natural language supervi- sion

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervi- sion. In International conference on machine learning, ...

    2021

  41. [49]

    https://sharegpt.com/, 2023

    ShareGPT. https://sharegpt.com/, 2023. 6

    2023

  42. [50]

    Self-distillation from the last mini-batch for consis- tency regularization

    Yiqing Shen, Liwu Xu, Yuzhe Yang, Yaqian Li, and Yandong Guo. Self-distillation from the last mini-batch for consis- tency regularization. In Proceedings of the IEEE/CVF con- ference on computer vision and pattern recognition , pages 11943–11952, 2022. 3

    2022

  43. [51]

    Towards vqa models that can read

    Amanpreet Singh, Vivek Natarajan, Meet Shah, Yu Jiang, Xinlei Chen, Dhruv Batra, Devi Parikh, and Marcus Rohrbach. Towards vqa models that can read. InProceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 8317–8326, 2019. 6

    2019

  44. [52]

    Emu: Generative pretraining in multimodality

    Quan Sun, Qiying Yu, Yufeng Cui, Fan Zhang, Xiaosong Zhang, Yueze Wang, Hongcheng Gao, Jingjing Liu, Tiejun Huang, and Xinlong Wang. Emu: Generative pretraining in multimodality. In The Twelfth International Conference on Learning Representations, 2023. 2, 6

    2023

  45. [53]

    Generative multimodal mod- els are in-context learners

    Quan Sun, Yufeng Cui, Xiaosong Zhang, Fan Zhang, Qiy- ing Yu, Yueze Wang, Yongming Rao, Jingjing Liu, Tiejun Huang, and Xinlong Wang. Generative multimodal mod- els are in-context learners. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , ...

    2024

  46. [54]

    Chameleon: Mixed-modal early-fusion foundation models

    Chameleon Team. Chameleon: Mixed-modal early-fusion foundation models. arXiv preprint arXiv:2405.09818, 2024. 2

    2024 arXiv

  47. [55]

    Gemma: Open models based on gemini research and tech- nology

    Gemma Team, Thomas Mesnard, Cassidy Hardin, Robert Dadashi, Surya Bhupatiraju, Shreya Pathak, Laurent Sifre, Morgane Rivi `ere, Mihir Sanjay Kale, Juliette Love, et al. Gemma: Open models based on gemini research and tech- nology. arXiv preprint arXiv:2403.08295, 2024. 8

    2024 arXiv

  48. [56]

    Neural discrete representation learning

    Aaron Van Den Oord, Oriol Vinyals, et al. Neural discrete representation learning. Advances in neural information pro- cessing systems, 30, 2017. 2

    2017

  49. [57]

    Qwen2-vl: Enhancing vision-language model’s perception of the world at any resolution

    Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, et al. Qwen2-vl: Enhancing vision-language model’s perception of the world at any resolution. arXiv preprint arXiv:2409.12191, 2024. 1, 2

    2024 arXiv

  50. [58]

    Cogvlm: Visual expert for pretrained language models

    Weihan Wang, Qingsong Lv, Wenmeng Yu, Wenyi Hong, Ji Qi, Yan Wang, Junhui Ji, Zhuoyi Yang, Lei Zhao, Xixuan Song, et al. Cogvlm: Visual expert for pretrained language models. arXiv preprint arXiv:2311.03079, 2023. 1

    2023 arXiv

  51. [59]

    Do llamas work in english? on the la- tent language of multilingual transformers

    Chris Wendler, Veniamin Veselovsky, Giovanni Monea, and Robert West. Do llamas work in english? on the la- tent language of multilingual transformers. arXiv preprint arXiv:2402.10588, 2024. 3, 4

    2024 arXiv

  52. [60]

    Libra: Building decoupled vision system on large lan- guage models

    Yifan Xu, Xiaoshan Yang, Yaguang Song, and Changsheng Xu. Libra: Building decoupled vision system on large lan- guage models. In ICML. OpenReview.net, 2024. 1

    2024

  53. [61]

    Qwen2 technical report.arXiv preprint arXiv:2407.10671, 2024

    An Yang, Baosong Yang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Zhou, Chengpeng Li, Chengyuan Li, Dayiheng Liu, Fei Huang, et al. Qwen2 technical report.arXiv preprint arXiv:2407.10671, 2024. 1

    2024 arXiv

  54. [62]

    Snapshot distillation: Teacher-student optimization in one generation

    Chenglin Yang, Lingxi Xie, Chi Su, and Alan L Yuille. Snapshot distillation: Teacher-student optimization in one generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 2859– 2868, 2019. 3

    2019

  55. [63]

    mplug-owl3: Towards long image-sequence understanding in multi-modal large language models

    Jiabo Ye, Haiyang Xu, Haowei Liu, Anwen Hu, Ming Yan, Qi Qian, Ji Zhang, Fei Huang, and Jingren Zhou. mplug-owl3: Towards long image-sequence understanding in multi-modal large language models. arXiv preprint arXiv:2408.04840, 2024. 1

    2024 arXiv

  56. [64]

    Language model beats diffusion–tokenizer is key to visual generation

    Lijun Yu, Jos ´e Lezama, Nitesh B Gundavarapu, Luca Ver- sari, Kihyuk Sohn, David Minnen, Yong Cheng, Vighnesh Birodkar, Agrim Gupta, Xiuye Gu, et al. Language model beats diffusion–tokenizer is key to visual generation. arXiv preprint arXiv:2310.05737, 2023. 2

    2023 arXiv

  57. [65]

    Mm-vet: Evaluating large multimodal models for integrated capabilities

    Weihao Yu, Zhengyuan Yang, Linjie Li, Jianfeng Wang, Kevin Lin, Zicheng Liu, Xinchao Wang, and Lijuan Wang. Mm-vet: Evaluating large multimodal models for integrated capabilities. arXiv preprint arXiv:2308.02490, 2023. 6

    2023 arXiv

  58. [66]

    How do large language models learn in-context? query and key matrices of in- context heads are two towers for metric learning

    Zeping Yu and Sophia Ananiadou. How do large language models learn in-context? query and key matrices of in- context heads are two towers for metric learning. arXiv preprint arXiv:2402.02872, 2024. 3, 4

    2024 arXiv

  59. [67]

    Transformer visualization via dictionary learning: con- textualized embedding as a linear superposition of trans- former factors

    Zeyu Yun, Yubei Chen, Bruno A Olshausen, and Yann Le- Cun. Transformer visualization via dictionary learning: con- textualized embedding as a linear superposition of trans- former factors. arXiv preprint arXiv:2103.15949, 2021. 3

    2021 arXiv

  60. [68]

    Sigmoid loss for language image pre-training

    Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, and Lucas Beyer. Sigmoid loss for language image pre-training. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 11975–11986, 2023. 1, 8

    2023

  61. [69]

    Be your own teacher: Improve the performance of convolutional neural networks via self distillation

    Linfeng Zhang, Jiebo Song, Anni Gao, Jingwei Chen, Chen- glong Bao, and Kaisheng Ma. Be your own teacher: Improve the performance of convolutional neural networks via self distillation. In Proceedings of the IEEE/CVF international conference on computer vision, pages 3713–3722...

    2019

  62. [70]

    Self- distillation: Towards efficient and compact neural networks

    Linfeng Zhang, Chenglong Bao, and Kaisheng Ma. Self- distillation: Towards efficient and compact neural networks. IEEE Transactions on Pattern Analysis and Machine Intelli- gence, 44(8):4388–4403, 2021. 3

    2021

  63. [71]

    Self-distillation as instance- specific label smoothing

    Zhilu Zhang and Mert Sabuncu. Self-distillation as instance- specific label smoothing. Advances in Neural Information Processing Systems, 33:2184–2195, 2020. 3

    2020

  64. [72]

    Decoupled knowledge distillation

    Borui Zhao, Quan Cui, Renjie Song, Yiyu Qiu, and Jiajun Liang. Decoupled knowledge distillation. In Proceedings of the IEEE/CVF Conference on computer vision and pattern recognition, pages 11953–11962, 2022. 3 11 BASIC: Boosting Visual Alignment with Intrinsic Refined Embeddin...

    2022

Pith tools

Reviewed August 5, 2026 · model on record in the stance chip above.